Problem Statement¶
Context¶
Businesses like banks which provide service have to worry about problem of 'Customer Churn' i.e. customers leaving and joining another service provider. It is important to understand which aspects of the service influence a customer's decision in this regard. Management can concentrate efforts on improvement of service, keeping in mind these priorities.
Objective¶
You as a Data scientist with the bank need to build a neural network based classifier that can determine whether a customer will leave the bank or not in the next 6 months.
Data Dictionary¶
CustomerId: Unique ID which is assigned to each customer
Surname: Last name of the customer
CreditScore: It defines the credit history of the customer.
Geography: A customerās location
Gender: It defines the Gender of the customer
Age: Age of the customer
Tenure: Number of years for which the customer has been with the bank
NumOfProducts: refers to the number of products that a customer has purchased through the bank.
Balance: Account balance
HasCrCard: It is a categorical variable which decides whether the customer has credit card or not.
EstimatedSalary: Estimated salary
isActiveMember: Is is a categorical variable which decides whether the customer is active member of the bank or not ( Active member in the sense, using bank products regularly, making transactions etc )
Exited : whether or not the customer left the bank within six month. It can take two values
** 0=No ( Customer did not leave the bank ) ** 1=Yes ( Customer left the bank )
# Installing the libraries with the specified version.
#!pip3 install tensorflow==2.15.0 scikit-learn==1.2.2 seaborn==0.13.1 matplotlib==3.7.1 numpy==1.25.2 pandas==2.0.3 imbalanced-learn==0.10.1 -q --user
#!pip3 install tensorflow==2.16.1
Importing necessary libraries¶
# Libraries to help with reading and manipulating data
import pandas as pd
import numpy as np
# libaries to help with data visualization
import matplotlib.pyplot as plt
import seaborn as sns
# Library to split data
from sklearn.model_selection import train_test_split
# library to import to standardize the data
from sklearn.preprocessing import StandardScaler, RobustScaler, LabelEncoder
# importing different functions to build models
import tensorflow as tf
from tensorflow import keras
from keras import backend
from keras.models import Sequential
from keras.layers import Dense, Dropout, BatchNormalization
# importing SMOTE
from imblearn.over_sampling import SMOTE
# importing metrics
from sklearn.metrics import confusion_matrix,roc_curve,classification_report,recall_score
import random
# Library to avoid the warnings
import warnings
warnings.filterwarnings("ignore")
2024-06-05 19:07:29.059233: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
Loading the dataset¶
#load the dataset
churn = pd.read_csv("churn.csv")
Data Overview¶
# view the first 5 rows of the data
churn.head()
| RowNumber | CustomerId | Surname | CreditScore | Geography | Gender | Age | Tenure | Balance | NumOfProducts | HasCrCard | IsActiveMember | EstimatedSalary | Exited | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | 15634602 | Hargrave | 619 | France | Female | 42 | 2 | 0.00 | 1 | 1 | 1 | 101348.88 | 1 |
| 1 | 2 | 15647311 | Hill | 608 | Spain | Female | 41 | 1 | 83807.86 | 1 | 0 | 1 | 112542.58 | 0 |
| 2 | 3 | 15619304 | Onio | 502 | France | Female | 42 | 8 | 159660.80 | 3 | 1 | 0 | 113931.57 | 1 |
| 3 | 4 | 15701354 | Boni | 699 | France | Female | 39 | 1 | 0.00 | 2 | 0 | 0 | 93826.63 | 0 |
| 4 | 5 | 15737888 | Mitchell | 850 | Spain | Female | 43 | 2 | 125510.82 | 1 | 1 | 1 | 79084.10 | 0 |
# view the last 5 rows of the data
churn.tail()
| RowNumber | CustomerId | Surname | CreditScore | Geography | Gender | Age | Tenure | Balance | NumOfProducts | HasCrCard | IsActiveMember | EstimatedSalary | Exited | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 9995 | 9996 | 15606229 | Obijiaku | 771 | France | Male | 39 | 5 | 0.00 | 2 | 1 | 0 | 96270.64 | 0 |
| 9996 | 9997 | 15569892 | Johnstone | 516 | France | Male | 35 | 10 | 57369.61 | 1 | 1 | 1 | 101699.77 | 0 |
| 9997 | 9998 | 15584532 | Liu | 709 | France | Female | 36 | 7 | 0.00 | 1 | 0 | 1 | 42085.58 | 1 |
| 9998 | 9999 | 15682355 | Sabbatini | 772 | Germany | Male | 42 | 3 | 75075.31 | 2 | 1 | 0 | 92888.52 | 1 |
| 9999 | 10000 | 15628319 | Walker | 792 | France | Female | 28 | 4 | 130142.79 | 1 | 1 | 0 | 38190.78 | 0 |
- Age, Balance, Tenure, CreditScore, and EstimatedSalary will probably have to be normalized.
- Geography and Gender will have to be one hot encoded to be of use in our model.
churn.shape
(10000, 14)
10000 rows and 14 columns
#Check the data types of the columns for the dataset
churn.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 10000 entries, 0 to 9999 Data columns (total 14 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 RowNumber 10000 non-null int64 1 CustomerId 10000 non-null int64 2 Surname 10000 non-null object 3 CreditScore 10000 non-null int64 4 Geography 10000 non-null object 5 Gender 10000 non-null object 6 Age 10000 non-null int64 7 Tenure 10000 non-null int64 8 Balance 10000 non-null float64 9 NumOfProducts 10000 non-null int64 10 HasCrCard 10000 non-null int64 11 IsActiveMember 10000 non-null int64 12 EstimatedSalary 10000 non-null float64 13 Exited 10000 non-null int64 dtypes: float64(2), int64(9), object(3) memory usage: 1.1+ MB
#Checking the Statistical Summary
churn.describe().T
| count | mean | std | min | 25% | 50% | 75% | max | |
|---|---|---|---|---|---|---|---|---|
| RowNumber | 10000.0 | 5.000500e+03 | 2886.895680 | 1.00 | 2500.75 | 5.000500e+03 | 7.500250e+03 | 10000.00 |
| CustomerId | 10000.0 | 1.569094e+07 | 71936.186123 | 15565701.00 | 15628528.25 | 1.569074e+07 | 1.575323e+07 | 15815690.00 |
| CreditScore | 10000.0 | 6.505288e+02 | 96.653299 | 350.00 | 584.00 | 6.520000e+02 | 7.180000e+02 | 850.00 |
| Age | 10000.0 | 3.892180e+01 | 10.487806 | 18.00 | 32.00 | 3.700000e+01 | 4.400000e+01 | 92.00 |
| Tenure | 10000.0 | 5.012800e+00 | 2.892174 | 0.00 | 3.00 | 5.000000e+00 | 7.000000e+00 | 10.00 |
| Balance | 10000.0 | 7.648589e+04 | 62397.405202 | 0.00 | 0.00 | 9.719854e+04 | 1.276442e+05 | 250898.09 |
| NumOfProducts | 10000.0 | 1.530200e+00 | 0.581654 | 1.00 | 1.00 | 1.000000e+00 | 2.000000e+00 | 4.00 |
| HasCrCard | 10000.0 | 7.055000e-01 | 0.455840 | 0.00 | 0.00 | 1.000000e+00 | 1.000000e+00 | 1.00 |
| IsActiveMember | 10000.0 | 5.151000e-01 | 0.499797 | 0.00 | 0.00 | 1.000000e+00 | 1.000000e+00 | 1.00 |
| EstimatedSalary | 10000.0 | 1.000902e+05 | 57510.492818 | 11.58 | 51002.11 | 1.001939e+05 | 1.493882e+05 | 199992.48 |
| Exited | 10000.0 | 2.037000e-01 | 0.402769 | 0.00 | 0.00 | 0.000000e+00 | 0.000000e+00 | 1.00 |
# check for missing values in the data, express as %
round(churn.isnull().sum() / churn.isnull().count() * 100, 2)
RowNumber 0.0 CustomerId 0.0 Surname 0.0 CreditScore 0.0 Geography 0.0 Gender 0.0 Age 0.0 Tenure 0.0 Balance 0.0 NumOfProducts 0.0 HasCrCard 0.0 IsActiveMember 0.0 EstimatedSalary 0.0 Exited 0.0 dtype: float64
- No missing values.
# check for duplicate values
churn.duplicated().sum()
0
- No duplicate values.
#Checking for unique values for each of the column
churn.nunique()
RowNumber 10000 CustomerId 10000 Surname 2932 CreditScore 460 Geography 3 Gender 2 Age 70 Tenure 11 Balance 6382 NumOfProducts 4 HasCrCard 2 IsActiveMember 2 EstimatedSalary 9999 Exited 2 dtype: int64
- Age has only 70 unique values, likely meaning large groupings of similar ages.
- Geography and Gender look to be categorical in nature.
- We have many continuous variables like CreditScore, Balance, EstimatedSalary, etc.
- RowNumber consists of uniques ID for each row and will not add value to the modeling
- CustomerId consists of uniques ID for customers and will not add value to the modeling
- Surname consists of last names customers and will not add value to this modeling
# Can see that several of these columns have limited data types, so would be better as categories
# Converting the data type of columns with category like data to 'category'
# don't convert Exited, since we will remove later and its the target var.
category_col = ['Geography','Gender']
churn[category_col] = churn[category_col].astype('category')
# take a look at the data types after conversion
churn.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 10000 entries, 0 to 9999 Data columns (total 14 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 RowNumber 10000 non-null int64 1 CustomerId 10000 non-null int64 2 Surname 10000 non-null object 3 CreditScore 10000 non-null int64 4 Geography 10000 non-null category 5 Gender 10000 non-null category 6 Age 10000 non-null int64 7 Tenure 10000 non-null int64 8 Balance 10000 non-null float64 9 NumOfProducts 10000 non-null int64 10 HasCrCard 10000 non-null int64 11 IsActiveMember 10000 non-null int64 12 EstimatedSalary 10000 non-null float64 13 Exited 10000 non-null int64 dtypes: category(2), float64(2), int64(9), object(1) memory usage: 957.4+ KB
# view the statistical summary of the non-numerical columns in the data
churn.describe(exclude=np.number).T
| count | unique | top | freq | |
|---|---|---|---|---|
| Surname | 10000 | 2932 | Smith | 32 |
| Geography | 10000 | 3 | France | 5014 |
| Gender | 10000 | 2 | Male | 5457 |
var_col = [
"CreditScore",
"Geography",
"Gender",
"Age",
"Tenure",
"Balance",
"NumOfProducts",
"HasCrCard",
"IsActiveMember",
"EstimatedSalary",
"Exited"
]
i=0
#summarize all values of categories
for column in var_col:
print(var_col[i])
print("-"*55)
print(churn[column].value_counts())
print("\n")
i+=1
CreditScore
-------------------------------------------------------
CreditScore
850 233
678 63
655 54
705 53
667 53
...
351 1
365 1
382 1
373 1
419 1
Name: count, Length: 460, dtype: int64
Geography
-------------------------------------------------------
Geography
France 5014
Germany 2509
Spain 2477
Name: count, dtype: int64
Gender
-------------------------------------------------------
Gender
Male 5457
Female 4543
Name: count, dtype: int64
Age
-------------------------------------------------------
Age
37 478
38 477
35 474
36 456
34 447
...
84 2
88 1
82 1
85 1
83 1
Name: count, Length: 70, dtype: int64
Tenure
-------------------------------------------------------
Tenure
2 1048
1 1035
7 1028
8 1025
5 1012
3 1009
4 989
9 984
6 967
10 490
0 413
Name: count, dtype: int64
Balance
-------------------------------------------------------
Balance
0.00 3617
105473.74 2
130170.82 2
130142.79 1
117419.35 1
...
88381.21 1
155060.41 1
57369.61 1
75075.31 1
150725.53 1
Name: count, Length: 6382, dtype: int64
NumOfProducts
-------------------------------------------------------
NumOfProducts
1 5084
2 4590
3 266
4 60
Name: count, dtype: int64
HasCrCard
-------------------------------------------------------
HasCrCard
1 7055
0 2945
Name: count, dtype: int64
IsActiveMember
-------------------------------------------------------
IsActiveMember
1 5151
0 4849
Name: count, dtype: int64
EstimatedSalary
-------------------------------------------------------
EstimatedSalary
24924.92 2
196526.55 1
158590.25 1
118913.53 1
38190.78 1
..
136869.31 1
113308.29 1
159334.93 1
20393.44 1
6847.73 1
Name: count, Length: 9999, dtype: int64
Exited
-------------------------------------------------------
Exited
0 7963
1 2037
Name: count, dtype: int64
churn_drop = churn.copy()
# RowNumber why?, CustomerId consists of uniques ID for customer, Surname may not be unique but not needed and will not add value to the modeling so delete.
churn_drop.drop(['RowNumber', 'CustomerId', 'Surname'], axis=1, inplace=True)
# view the first 5 rows of the data
churn_drop.head()
| CreditScore | Geography | Gender | Age | Tenure | Balance | NumOfProducts | HasCrCard | IsActiveMember | EstimatedSalary | Exited | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 619 | France | Female | 42 | 2 | 0.00 | 1 | 1 | 1 | 101348.88 | 1 |
| 1 | 608 | Spain | Female | 41 | 1 | 83807.86 | 1 | 0 | 1 | 112542.58 | 0 |
| 2 | 502 | France | Female | 42 | 8 | 159660.80 | 3 | 1 | 0 | 113931.57 | 1 |
| 3 | 699 | France | Female | 39 | 1 | 0.00 | 2 | 0 | 0 | 93826.63 | 0 |
| 4 | 850 | Spain | Female | 43 | 2 | 125510.82 | 1 | 1 | 1 | 79084.10 | 0 |
Exploratory Data Analysis¶
# Number each graph/figure start with 1
figNo=1
#Make it easier to see data by update to Yes/No and Categorical for visualization
churn_visual = churn_drop.copy()
churn_visual = churn_visual.replace({'Exited':{0:'No', 1:'Yes'},
'HasCrCard':{0:'No',1:'Yes'},
'IsActiveMember':{0:'No',1:'Yes'}}) #just these for viz
Univariate Analysis¶
# function to plot a boxplot and a histogram along the same scale.
def histogram_boxplot(data, feature, figsize=(12, 7), kde=False, bins=None):
"""
Boxplot and histogram combined
data: dataframe
feature: dataframe column
figsize: size of figure (default (12,7))
kde: whether to show the density curve (default False)
bins: number of bins for histogram (default None)
"""
f2, (ax_box2, ax_hist2) = plt.subplots(
nrows=2, # Number of rows of the subplot grid= 2
sharex=True, # x-axis will be shared among all subplots
gridspec_kw={"height_ratios": (0.25, 0.75)},
figsize=figsize,
) # creating the 2 subplots
sns.boxplot(
data=data, x=feature, ax=ax_box2, showmeans=True, color="violet"
) # boxplot will be created and a star will indicate the mean value of the column
sns.histplot(
data=data, x=feature, kde=kde, ax=ax_hist2, bins=bins, palette="winter"
) if bins else sns.histplot(
data=data, x=feature, kde=kde, ax=ax_hist2
) # For histogram
ax_hist2.axvline(
data[feature].mean(), color="green", linestyle="--"
) # Add mean to the histogram
ax_hist2.axvline(
data[feature].median(), color="black", linestyle="-"
) # Add median to the histogram
# function to create labeled barplots
def labeled_barplot(data, feature, perc=False, n=None):
"""
Barplot with percentage at the top
data: dataframe
feature: dataframe column
perc: whether to display percentages instead of count (default is False)
n: displays the top n category levels (default is None, i.e., display all levels)
"""
total = len(data[feature]) # length of the column
count = data[feature].nunique()
if n is None:
plt.figure(figsize=(count + 1, 5))
else:
plt.figure(figsize=(n + 1, 5))
plt.xticks(rotation=90, fontsize=15)
ax = sns.countplot(
data=data,
x=feature,
palette="Paired",
order=data[feature].value_counts().index[:n].sort_values(),
)
for p in ax.patches:
if perc == True:
label = "{:.1f}%".format(
100 * p.get_height() / total
) # percentage of each class of the category
else:
label = p.get_height() # count of each level of the category
x = p.get_x() + p.get_width() / 2 # width of the plot
y = p.get_height() # height of the plot
ax.annotate(
label,
(x, y),
ha="center",
va="center",
size=12,
xytext=(0, 5),
textcoords="offset points",
) # annotate the percentage
plt.show() # show the plot
Observations on CreditScore¶
Title = 'CreditScore'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1
histogram_boxplot(churn_visual, Title)
Fig 1 - CreditScore
Observations¶
CreditScoredistribution is left skewed.- The plot shows outliers below 400.
- The Median is about 660 and the mean is very close to the median
- There are a large number of customers with
CreditScoreabove 840. - These values appear valid due to wide variations in credit scores and thus will not be adjusted.
Observations on Age¶
Title = 'Age'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1
histogram_boxplot(churn_visual, Title)
Fig 2 - Age
Observations¶
- The distribution of
Ageis skewed to the right. - The plot shows outliers at the right side (high end of age) above 60.
- The median is about 37 and the mean is about 39.
- We will not adjust these outliers as they represent real ages and trends in age.
Observations on Balance¶
Title = 'Balance'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1
histogram_boxplot(churn_visual, Title)
Fig 3 - Balance
Observations¶
- The distribution of
Balanceis would be normal if not for the huge number of customers with 0Balance(3617). - This may indicate customers with a credit card only and not a bank account.
- No outliers.
Observations on Estimated Salary¶
Title = 'EstimatedSalary'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1
histogram_boxplot(churn_visual, Title)
Fig 4 - EstimatedSalary
Observations¶
- The distribution of
EstimatedSalaryis close to even across the range. - The plot shows no outliers.
- This variable will likely have little influence.
Observations on Exited¶
Title = 'Exited'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1
labeled_barplot(churn_visual, Title, perc=True)
Fig 5 - Exited
Observations¶
- Almost 80% of customers churned.
- The data set is imbalanced.
Observations on Geography¶
Title = 'Geography'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1
labeled_barplot(churn_visual, Title, perc=True)
Fig 6 - Geography
Observations¶
- 50% of customers are in France.
- 25% are in Germany and 25% are in Spain.
Observations on Gender¶
Title = 'Gender'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1
labeled_barplot(churn_visual, Title, perc=True)
Fig 7 - Gender
Observations¶
- Almost 55% of customers are male.
Observations on Tenure¶
Title = 'Tenure'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1
labeled_barplot(churn_visual, Title, perc=True)
Fig 8 - Tenure
Observations¶
- The distribution of
Tenureis fairly even between 1-9 years. - Similarly, customers at the low end (0) and high end (10) are about the same distribution.
- This data is strange in that it appears there hasn't been much growth this year in customers, and no one over 10 years.
Observations on Number of Products¶
Title = 'NumOfProducts'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1
labeled_barplot(churn_visual, Title, perc=True)
Fig 9 - NumOfProducts
Observations¶
- Around 51% of all customers have only 1 product, while ~46% have 2.
- Those with more than 2 are a very small minority.
Observations on Has Credit Card¶
Title = 'HasCrCard'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1
labeled_barplot(churn_visual, Title, perc=True)
Fig 10 - HasCrCard
Observations¶
- 70.5% of customers have a credit card.
Observations on Is Active Member¶
Title = 'IsActiveMember'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1
labeled_barplot(churn_visual, Title, perc=True)
Fig 11 - IsActiveMember
Observations¶
- Almost 52% of customers are active.
Bivariate Analysis¶
# function to plot stacked bar chart
def stacked_barplot(data, predictor, target):
"""
Print the category counts and plot a stacked bar chart
data: dataframe
predictor: independent variable
target: target variable
"""
count = data[predictor].nunique()
sorter = data[target].value_counts().index[-1]
tab1 = pd.crosstab(data[predictor], data[target], margins=True).sort_values(
by=sorter, ascending=False
)
print(tab1)
print("-" * 120)
tab = pd.crosstab(data[predictor], data[target], normalize="index").sort_values(
by=sorter, ascending=False
)
tab.plot(kind="bar", stacked=True, figsize=(count + 1, 5))
plt.legend(
loc="lower left",
frameon=False,
)
plt.legend(loc="upper left", bbox_to_anchor=(1, 1))
plt.show()# function to plot stacked bar chart
print("Fig "+str(figNo)+" - "+ "Pairplot")
figNo+=1
churn_pair = churn.copy()
churn_pair = churn_pair.replace({'Exited':{0:'No', 1:'Yes'}}) # do this so all data viz can benefit
sns.pairplot(churn_pair, diag_kind="kde",kind="scatter", hue="Exited", dropna=True, corner=True)
plt.show()
Fig 12 - Pairplot
Observations¶
CreditScorefor churn/no churn have a similar distribution.Ageshows that middle-aged customers are more likely to churn than younger or older customers.Tenureshows similar churn/no churn distribution.Balanceshows that smaller balances churn less than larger balances.NumOfProductsshows that less products churn more than more products.HasCrCardshows similar distributions for churn/no churn.IsActiveMembershows that Active members churn less than non active members.EstimatedSalaryshows similar distributions for churn/no churn customers.
Correlation plot¶
Title = 'Correlation Plot'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1
plt.figure(figsize=(15, 7))
sns.heatmap(churn_visual.corr(numeric_only=True), annot=True, vmin=-1, vmax=1, fmt=".2f", cmap="Spectral")
plt.show()
Fig 13 - Correlation Plot
Observations¶
- None of these items are strongly correlated.
- Balance and NumOfProducts is slighly negatively correlated.
- Not much insight to be gained here.
### Function to plot distributions
def distribution_plot_wrt_target(data, predictor, target):
fig, axs = plt.subplots(2, 2, figsize=(12, 10))
target_uniq = data[target].unique()
axs[0, 0].set_title("Distribution of target for target=" + str(target_uniq[0]))
sns.histplot(
data=data[data[target] == target_uniq[0]],
x=predictor,
kde=True,
ax=axs[0, 0],
color="teal",
)
axs[0, 1].set_title("Distribution of target for target=" + str(target_uniq[1]))
sns.histplot(
data=data[data[target] == target_uniq[1]],
x=predictor,
kde=True,
ax=axs[0, 1],
color="orange",
)
axs[1, 0].set_title("Boxplot w.r.t target")
sns.boxplot(data=data, x=target, y=predictor, ax=axs[1, 0], palette="gist_rainbow")
axs[1, 1].set_title("Boxplot (without outliers) w.r.t target")
sns.boxplot(
data=data,
x=target,
y=predictor,
ax=axs[1, 1],
showfliers=False,
palette="gist_rainbow",
)
plt.tight_layout()
plt.show()
Exited Vs Geography¶
Title = 'Exited Vs Geography'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1
stacked_barplot(churn_visual, "Geography", "Exited" )
Fig 14 - Exited Vs Geography Exited No Yes All Geography All 7963 2037 10000 Germany 1695 814 2509 France 4204 810 5014 Spain 2064 413 2477 ------------------------------------------------------------------------------------------------------------------------
Observations¶
- Spanish customers churn similarly in proportion to French customers
- German customers churn at a higher ration, but also in overall number.
Exited Vs Gender¶
Title = 'Exited Vs Gender'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1
stacked_barplot(churn_visual, "Gender", "Exited")
Fig 15 - Exited Vs Gender Exited No Yes All Gender All 7963 2037 10000 Female 3404 1139 4543 Male 4559 898 5457 ------------------------------------------------------------------------------------------------------------------------
Observations¶
- Females churn at a higher proportion than males and represent more of the overall churned customers.
Exited Vs Has Credit Card¶
Title = 'Exited Vs Has Credit Card'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1
stacked_barplot(churn_visual, "HasCrCard", "Exited")
Fig 16 - Exited Vs Has Credit Card Exited No Yes All HasCrCard All 7963 2037 10000 Yes 5631 1424 7055 No 2332 613 2945 ------------------------------------------------------------------------------------------------------------------------
Observations¶
- Customers with or without credit cards churn at about the same proportion.
- Customers with a credit card make up the bulk of churners.
Exited Vs Is active member¶
Title = 'Exited Vs Is active member'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1
stacked_barplot(churn_visual, "IsActiveMember", "Exited")
Fig 17 - Exited Vs Is active member Exited No Yes All IsActiveMember All 7963 2037 10000 No 3547 1302 4849 Yes 4416 735 5151 ------------------------------------------------------------------------------------------------------------------------
Observations¶
- Active customers churn less proportionally.
- Non-active customers make up the bulk of churners.
Exited Vs Credit Score¶
Title = 'Exited Vs Credit Score'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1
distribution_plot_wrt_target(churn_visual, "CreditScore", "Exited")
Fig 18 - Exited Vs Credit Score
Observations¶
- The distribution of
CreditScoreis is similar for both groups of customers. - There are several outliers with
CreditScorebelow 400 that have churned. - We will not adjust these outliers as they represent real trends.
Exited Vs Age¶
Title = 'Exited Vs Age'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1
distribution_plot_wrt_target(churn_visual, "Age", "Exited")
Fig 19 - Exited Vs Age
Observations¶
- The plot shows many outliers at the higher end of age
- Customers between 40-50 show a tendency to churn.
- Customers between 30-40 and above 60 show low tendency to churn.
- We will not adjust these outliers as they represent real ages and trends in age.
Exited Vs Tenure¶
Title = 'Exited Vs Tenure'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1
distribution_plot_wrt_target(churn_visual, "Tenure", "Exited")
Fig 20 - Exited Vs Tenure
Observations¶
- No outliers.
- Distribution is similar in years 1-9, and very low in 0+10
Exited Vs Balance¶
Title = 'Exited Vs Balance'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1
distribution_plot_wrt_target(churn_visual, "Balance", "Exited")
Fig 21 - Exited Vs Balance
Observations¶
- Customers with a lower
Balancetend to churn more.
Exited Vs Number of Products¶
Title = 'Exited Vs Number of Products'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1
distribution_plot_wrt_target(churn_visual, "NumOfProducts", "Exited")
Fig 22 - Exited Vs Number of Products
Observations¶
- Customers with 1 product churn the most. Since they only have 1 product, they probably have little incentive not to churn.
- Customers with 2 products churn the least.
Exited Vs Estimated Salary¶
Title = 'Exited Vs Estimated Salary'
print("Fig "+ str(figNo)+" - "+ Title)
figNo+=1
distribution_plot_wrt_target(churn_visual, "EstimatedSalary", "Exited")
Fig 23 - Exited Vs Estimated Salary
Observations¶
- The distribution of
EstimatedSalaryis similar across both groups. - This variable will likely not influence much in the model.
Data Preprocessing¶
#Copy churn_data
churn_data = churn_drop.copy()
Dummy Variable Creation¶
# Creating dummy variables for categorical variables
# churn_data = pd.get_dummies(data=churn_data, drop_first=True)
dummies = pd.get_dummies(churn_data[['Geography', 'Gender']], drop_first=True)
churn_data = pd.concat([churn_data.drop(['Geography', 'Gender'],axis=1), dummies],axis=1)
churn_data.head()
| CreditScore | Age | Tenure | Balance | NumOfProducts | HasCrCard | IsActiveMember | EstimatedSalary | Exited | Geography_Germany | Geography_Spain | Gender_Male | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 619 | 42 | 2 | 0.00 | 1 | 1 | 1 | 101348.88 | 1 | False | False | False |
| 1 | 608 | 41 | 1 | 83807.86 | 1 | 0 | 1 | 112542.58 | 0 | False | True | False |
| 2 | 502 | 42 | 8 | 159660.80 | 3 | 1 | 0 | 113931.57 | 1 | False | False | False |
| 3 | 699 | 39 | 1 | 0.00 | 2 | 0 | 0 | 93826.63 | 0 | False | False | False |
| 4 | 850 | 43 | 2 | 125510.82 | 1 | 1 | 1 | 79084.10 | 0 | False | True | False |
churn_data.shape
(10000, 12)
Data Normalization¶
churn_data_scaled=churn_data.copy()
churn_data_scaled.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 10000 entries, 0 to 9999 Data columns (total 12 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 CreditScore 10000 non-null int64 1 Age 10000 non-null int64 2 Tenure 10000 non-null int64 3 Balance 10000 non-null float64 4 NumOfProducts 10000 non-null int64 5 HasCrCard 10000 non-null int64 6 IsActiveMember 10000 non-null int64 7 EstimatedSalary 10000 non-null float64 8 Exited 10000 non-null int64 9 Geography_Germany 10000 non-null bool 10 Geography_Spain 10000 non-null bool 11 Gender_Male 10000 non-null bool dtypes: bool(3), float64(2), int64(7) memory usage: 732.6 KB
# defining the list of numerical columns
cols_list = ["CreditScore","Age","Balance","EstimatedSalary","NumOfProducts","Tenure"]
# creating an instance of the RobustScaler since it will work better with outliers than standard
scaler = RobustScaler()
#X_train[cols_list] = sc.fit_transform(X_train[cols_list])
churn_data_scaled[cols_list] = scaler.fit_transform(churn_data_scaled[cols_list])
#Printing the maximum value and the minimum value of the independent variable.
print(churn_data_scaled.max())
print(churn_data_scaled.min())
CreditScore 1.477612 Age 4.583333 Tenure 1.25 Balance 1.204124 NumOfProducts 3.0 HasCrCard 1 IsActiveMember 1 EstimatedSalary 1.014356 Exited 1 Geography_Germany True Geography_Spain True Gender_Male True dtype: object CreditScore -2.253731 Age -1.583333 Tenure -1.25 Balance -0.76148 NumOfProducts 0.0 HasCrCard 0 IsActiveMember 0 EstimatedSalary -1.018257 Exited 0 Geography_Germany False Geography_Spain False Gender_Male False dtype: object
churn_data_scaled.shape
(10000, 12)
churn_data_scaled.head()
| CreditScore | Age | Tenure | Balance | NumOfProducts | HasCrCard | IsActiveMember | EstimatedSalary | Exited | Geography_Germany | Geography_Spain | Gender_Male | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | -0.246269 | 0.416667 | -0.75 | -0.761480 | 0.0 | 1 | 1 | 0.011739 | 1 | False | False | False |
| 1 | -0.328358 | 0.333333 | -1.00 | -0.104906 | 0.0 | 0 | 1 | 0.125512 | 0 | False | True | False |
| 2 | -1.119403 | 0.416667 | 0.75 | 0.489346 | 2.0 | 1 | 0 | 0.139630 | 1 | False | False | False |
| 3 | 0.350746 | 0.166667 | -1.00 | -0.761480 | 1.0 | 0 | 0 | -0.064717 | 0 | False | False | False |
| 4 | 1.477612 | 0.500000 | -0.75 | 0.221806 | 0.0 | 1 | 1 | -0.214561 | 0 | False | True | False |
Train-validation-test Split¶
X = churn_data_scaled.drop(['Exited'],axis=1)
y = churn_data_scaled['Exited']
# Splitting the dataset into the Training and Testing set.
X_large, X_test, y_large, y_test = train_test_split(X, y, test_size = 0.2, random_state = 42,stratify=y,shuffle = True)
# Splitting the dataset into the Training and Validation set.
X_train, X_val, y_train, y_val = train_test_split(X_large, y_large, test_size = 0.25, random_state = 42,stratify=y_large, shuffle = True)
print(X_train.shape, X_val.shape, X_test.shape)
(6000, 11) (2000, 11) (2000, 11)
print(y_train.shape, y_val.shape, y_test.shape)
(6000,) (2000,) (2000,)
X_train.head()
| CreditScore | Age | Tenure | Balance | NumOfProducts | HasCrCard | IsActiveMember | EstimatedSalary | Geography_Germany | Geography_Spain | Gender_Male | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1995 | -0.507463 | 0.583333 | 0.00 | -0.011961 | 1.0 | 1 | 1 | 0.064755 | False | False | False |
| 2724 | -1.485075 | 0.083333 | 0.75 | 0.183515 | 0.0 | 1 | 0 | 0.299880 | True | False | False |
| 5224 | 1.126866 | 0.500000 | -0.50 | -0.761480 | 0.0 | 1 | 0 | -0.286041 | False | True | True |
| 7697 | -0.380597 | 0.333333 | -0.50 | -0.761480 | 1.0 | 1 | 0 | -0.466032 | False | True | False |
| 1226 | -0.902985 | 0.416667 | 0.25 | -0.069523 | 1.0 | 1 | 0 | -0.253543 | True | False | False |
X_val.head()
| CreditScore | Age | Tenure | Balance | NumOfProducts | HasCrCard | IsActiveMember | EstimatedSalary | Geography_Germany | Geography_Spain | Gender_Male | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 6263 | -1.544776 | 0.000000 | -0.50 | -0.761480 | 1.0 | 1 | 1 | 0.811278 | False | False | True |
| 7644 | 0.171642 | -0.750000 | 1.00 | -0.761480 | 0.0 | 1 | 0 | 0.344734 | False | False | True |
| 429 | -0.626866 | 0.250000 | -1.00 | 0.016327 | 0.0 | 0 | 0 | 0.349714 | True | False | True |
| 647 | -0.552239 | 0.083333 | 0.50 | -0.117038 | 0.0 | 1 | 0 | -0.926929 | False | False | True |
| 8353 | -0.955224 | -0.416667 | 0.25 | -0.761480 | 0.0 | 1 | 1 | 0.332038 | False | True | False |
X_test.head()
| CreditScore | Age | Tenure | Balance | NumOfProducts | HasCrCard | IsActiveMember | EstimatedSalary | Geography_Germany | Geography_Spain | Gender_Male | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 5702 | -0.500000 | -0.083333 | 0.50 | -0.761480 | 1.0 | 1 | 0 | -0.060078 | False | False | True |
| 3667 | -0.947761 | -0.333333 | -0.25 | 0.264996 | 1.0 | 0 | 0 | -0.458611 | True | False | True |
| 1617 | -0.708955 | 0.250000 | -0.25 | -0.761480 | 1.0 | 0 | 1 | 0.053256 | False | True | False |
| 5673 | -0.097015 | -0.250000 | 0.00 | 0.330564 | 1.0 | 0 | 0 | -0.673305 | False | True | True |
| 4272 | -0.089552 | -0.250000 | -0.50 | -0.151764 | 0.0 | 1 | 1 | 0.694721 | False | True | False |
print(y_val.max())
print(y_val.min())
1 0
print(X_train.shape[0], "train samples")
print(X_val.shape[0], "validation samples")
print(X_test.shape[0], "test samples")
6000 train samples 2000 validation samples 2000 test samples
Model Building¶
Model Evaluation Criterion¶
Our model needs to predict customer churn.
The nature of predictions made by the classification model will translate as follows:
True positives (TP) are failures correctly predicted by the model.
True negatives (TN) are correctly predicted by the model.
False negatives (FN) are real failures where it is not detected by the model.
False positives (FP) are failures detected by the model when it should not have been.
accuracy - how many of our predictions were true
precision - of our postive predictions, how many were true?
recall - out of all the ones that should be true, how many did we correctly predict
f1 - how effective we make the tradeoff between precision and recall
Since a bank wants to detect customers who are at risk of leaving in the next 6 months, they would want to focus on recall to maximize detection of customers at risk and avoid false negatives.
# defining the batch size and # epochs upfront
epochs = 50
batch_size = 64
Let's create a function for plotting the confusion matrix
def make_confusion_matrix(actual_targets, predicted_targets):
"""
To plot the confusion_matrix with percentages
actual_targets: actual target (dependent) variable values
predicted_targets: predicted target (dependent) variable values
"""
cm = confusion_matrix(actual_targets, predicted_targets)
group_names = ['True Positive','False Negative','False Positive','True Negative']
categories = [ 'Not Exiting','Exiting']
group_labels = ["{}\n".format(value) for value in group_names]
group_counts = ["{0:0.0f}\n".format(value) for value in cm.flatten()]
group_percentages = ["{0:.2%}".format(value) for value in cm.flatten()/np.sum(cm)]
box_labels = [f"{v1}{v2}{v3}".strip() for v1, v2, v3 in zip(group_labels,group_counts,group_percentages)]
box_labels = np.asarray(box_labels).reshape(cm.shape[0],cm.shape[1])
accuracy = np.trace(cm) / float(np.sum(cm))
precision = cm[1,1] / sum(cm[:,1])
recall = cm[1,1] / sum(cm[1,:])
f1_score = 2*precision*recall / (precision + recall)
stats_text = "\n\nAccuracy={:0.3f}\nPrecision={:0.3f}\nRecall={:0.3f}\nF1 Score={:0.3f}".format(
accuracy,precision,recall,f1_score)
plt.figure(figsize=(6, 4))
sns.heatmap(cm, annot=box_labels,fmt="",cmap='Blues',xticklabels=categories,yticklabels=categories)
categories = [ 'Not Exiting','Exiting']
plt.ylabel("True label")
plt.xlabel('Predicted label' + stats_text)
def plot_train_vs_val_loss(loss, val_loss):
"""
Plotting Train Loss vs Validation Loss
loss: saved loss values from each epoch
val_loss: saved validation loss values from each epoch
"""
plt.plot(loss)
plt.plot(val_loss)
plt.title('model loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
def plot_train_vs_val_recall(recall, val_recall):
"""
Plotting Train recall vs Validation recall
recall: saved recall values from each epoch
val_recall: saved validation recall values from each epoch
"""
plt.plot(recall)
plt.plot(val_recall)
plt.title('model recall')
plt.ylabel('recall')
plt.xlabel('Epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
# defining a function to compute different metrics to check performance of a classification model built using statsmodels
def model_performance_classification(
model, predictors, target, threshold=0.5
):
"""
Function to compute different metrics to check classification model performance
model: classifier
predictors: independent variables
target: dependent variable
threshold: threshold for classifying the observation as class 1
"""
# checking which probabilities are greater than threshold
pred = model.predict(predictors) > threshold
# pred_temp = model.predict(predictors) > threshold
# # rounding off the above values to get classes
# pred = np.round(pred_temp)
acc = accuracy_score(target, pred) # to compute Accuracy
recall = recall_score(target, pred, average='weighted') # to compute Recall
precision = precision_score(target, pred, average='weighted') # to compute Precision
f1 = f1_score(target, pred, average='weighted') # to compute F1-score
# creating a dataframe of metrics
df_perf = pd.DataFrame(
{"Accuracy": acc, "Recall": recall, "Precision": precision, "F1 Score": f1,},
index=[0],
)
return df_perf
#Save metrics for later comparison, since we are optimizing for recall, we'll save recall
training_metrics = pd.DataFrame(columns=["recall"])
validation_metrics = pd.DataFrame(columns=["recall"])
Neural Network (64,32,16,8) with SGD Optimizer¶
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)
model_name = "Neural Network (64,32,16,8) with SGD"
#Initializing the neural network
sgd_model = Sequential()
sgd_model.add(Dense(64, activation='relu', input_dim = X_train.shape[1]))
sgd_model.add(Dense(32, activation='relu'))
sgd_model.add(Dense(16, activation='relu'))
sgd_model.add(Dense(8, activation='relu'))
sgd_model.add(Dense(1, activation = 'sigmoid'))
# use SGD as the optimizer.
optimizer = tf.keras.optimizers.SGD()
# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
check if this is the correct metric¶
## compile the model with binary cross entropy as loss function and recall as the metric.
sgd_model.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
sgd_model.summary()
Model: "sequential"
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāā ā Layer (type) ā Output Shape ā Param # ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā dense (Dense) ā (None, 64) ā 768 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_1 (Dense) ā (None, 32) ā 2,080 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_2 (Dense) ā (None, 16) ā 528 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_3 (Dense) ā (None, 8) ā 136 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_4 (Dense) ā (None, 1) ā 9 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāā
Total params: 3,521 (13.75 KB)
Trainable params: 3,521 (13.75 KB)
Non-trainable params: 0 (0.00 B)
check batch size and epochs¶
# Fitting the ANN
sgd_history = sgd_model.fit(
X_train, y_train,
batch_size=batch_size,
epochs=epochs,
validation_data=(X_val,y_val),
verbose=1
)
Epoch 1/50 94/94 āāāāāāāāāāāāāāāāāāāā 1s 3ms/step - loss: 0.6280 - recall: 0.0536 - val_loss: 0.5475 - val_recall: 0.0000e+00 Epoch 2/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5328 - recall: 0.0000e+00 - val_loss: 0.5128 - val_recall: 0.0000e+00 Epoch 3/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5050 - recall: 0.0000e+00 - val_loss: 0.4996 - val_recall: 0.0000e+00 Epoch 4/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4929 - recall: 0.0000e+00 - val_loss: 0.4897 - val_recall: 0.0000e+00 Epoch 5/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4834 - recall: 0.0000e+00 - val_loss: 0.4802 - val_recall: 0.0000e+00 Epoch 6/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4739 - recall: 0.0000e+00 - val_loss: 0.4701 - val_recall: 0.0000e+00 Epoch 7/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4638 - recall: 0.0000e+00 - val_loss: 0.4603 - val_recall: 0.0000e+00 Epoch 8/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4541 - recall: 0.0000e+00 - val_loss: 0.4514 - val_recall: 0.0000e+00 Epoch 9/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4455 - recall: 0.0000e+00 - val_loss: 0.4444 - val_recall: 0.0000e+00 Epoch 10/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4388 - recall: 0.0000e+00 - val_loss: 0.4392 - val_recall: 0.0000e+00 Epoch 11/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4337 - recall: 0.0000e+00 - val_loss: 0.4355 - val_recall: 0.0000e+00 Epoch 12/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4299 - recall: 0.0000e+00 - val_loss: 0.4328 - val_recall: 0.0000e+00 Epoch 13/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4272 - recall: 8.9250e-04 - val_loss: 0.4308 - val_recall: 0.0025 Epoch 14/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4250 - recall: 0.0036 - val_loss: 0.4292 - val_recall: 0.0074 Epoch 15/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4232 - recall: 0.0166 - val_loss: 0.4278 - val_recall: 0.0123 Epoch 16/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4216 - recall: 0.0231 - val_loss: 0.4265 - val_recall: 0.0344 Epoch 17/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4201 - recall: 0.0450 - val_loss: 0.4253 - val_recall: 0.0516 Epoch 18/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4186 - recall: 0.0706 - val_loss: 0.4242 - val_recall: 0.0713 Epoch 19/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4173 - recall: 0.0923 - val_loss: 0.4232 - val_recall: 0.0885 Epoch 20/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4161 - recall: 0.0992 - val_loss: 0.4221 - val_recall: 0.1081 Epoch 21/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4148 - recall: 0.1107 - val_loss: 0.4210 - val_recall: 0.1302 Epoch 22/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4136 - recall: 0.1321 - val_loss: 0.4199 - val_recall: 0.1499 Epoch 23/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4123 - recall: 0.1580 - val_loss: 0.4188 - val_recall: 0.1646 Epoch 24/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.4111 - recall: 0.1710 - val_loss: 0.4177 - val_recall: 0.1744 Epoch 25/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4098 - recall: 0.1894 - val_loss: 0.4165 - val_recall: 0.1843 Epoch 26/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4085 - recall: 0.2009 - val_loss: 0.4153 - val_recall: 0.1966 Epoch 27/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4072 - recall: 0.2161 - val_loss: 0.4140 - val_recall: 0.2039 Epoch 28/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4058 - recall: 0.2279 - val_loss: 0.4127 - val_recall: 0.2088 Epoch 29/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4043 - recall: 0.2404 - val_loss: 0.4113 - val_recall: 0.2138 Epoch 30/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4028 - recall: 0.2485 - val_loss: 0.4098 - val_recall: 0.2310 Epoch 31/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4012 - recall: 0.2551 - val_loss: 0.4083 - val_recall: 0.2383 Epoch 32/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3995 - recall: 0.2672 - val_loss: 0.4067 - val_recall: 0.2359 Epoch 33/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3977 - recall: 0.2760 - val_loss: 0.4050 - val_recall: 0.2457 Epoch 34/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3958 - recall: 0.2833 - val_loss: 0.4033 - val_recall: 0.2531 Epoch 35/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3938 - recall: 0.2957 - val_loss: 0.4014 - val_recall: 0.2654 Epoch 36/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3918 - recall: 0.3055 - val_loss: 0.3995 - val_recall: 0.2727 Epoch 37/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3897 - recall: 0.3043 - val_loss: 0.3974 - val_recall: 0.2801 Epoch 38/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3875 - recall: 0.3174 - val_loss: 0.3953 - val_recall: 0.2850 Epoch 39/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3852 - recall: 0.3208 - val_loss: 0.3932 - val_recall: 0.2948 Epoch 40/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3829 - recall: 0.3296 - val_loss: 0.3910 - val_recall: 0.3022 Epoch 41/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3806 - recall: 0.3392 - val_loss: 0.3889 - val_recall: 0.3145 Epoch 42/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3783 - recall: 0.3489 - val_loss: 0.3867 - val_recall: 0.3170 Epoch 43/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3760 - recall: 0.3515 - val_loss: 0.3844 - val_recall: 0.3268 Epoch 44/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3737 - recall: 0.3598 - val_loss: 0.3821 - val_recall: 0.3415 Epoch 45/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3714 - recall: 0.3702 - val_loss: 0.3797 - val_recall: 0.3538 Epoch 46/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3691 - recall: 0.3779 - val_loss: 0.3775 - val_recall: 0.3661 Epoch 47/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3668 - recall: 0.3848 - val_loss: 0.3753 - val_recall: 0.3710 Epoch 48/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3645 - recall: 0.3871 - val_loss: 0.3732 - val_recall: 0.3833 Epoch 49/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3623 - recall: 0.3965 - val_loss: 0.3713 - val_recall: 0.3857 Epoch 50/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3603 - recall: 0.4028 - val_loss: 0.3695 - val_recall: 0.3980
Loss function
#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(loss=sgd_history.history['loss'],val_loss=sgd_history.history['val_loss'])
- The loss in the validation and train sets is close and shows little noise
Recall
#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(sgd_history.history['recall'],sgd_history.history['val_recall'])
#Predicting the results using best as a threshold
y_train_pred = sgd_model.predict(X_train)
y_train_pred = (y_train_pred > 0.5)
y_train_pred
188/188 āāāāāāāāāāāāāāāāāāāā 0s 750us/step
array([[False],
[False],
[False],
...,
[ True],
[False],
[ True]])
#Predicting the results using best as a threshold
y_val_pred = sgd_model.predict(X_val)
y_val_pred = (y_val_pred > 0.5)
y_val_pred
63/63 āāāāāāāāāāāāāāāāāāāā 0s 542us/step
array([[False],
[False],
[ True],
...,
[False],
[False],
[ True]])
#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train, y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val, y_val_pred)
print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])
recall 0.429272 Name: Neural Network (64,32,16,8) with SGD, dtype: float64 recall 0.398034 Name: Neural Network (64,32,16,8) with SGD, dtype: float64
Classification report
#classification report
print(classification_report(y_train, y_train_pred))
precision recall f1-score support
0 0.87 0.97 0.91 4777
1 0.76 0.43 0.55 1223
accuracy 0.86 6000
macro avg 0.81 0.70 0.73 6000
weighted avg 0.85 0.86 0.84 6000
#classification report
print(classification_report(y_val, y_val_pred))
precision recall f1-score support
0 0.86 0.97 0.91 1593
1 0.75 0.40 0.52 407
accuracy 0.85 2000
macro avg 0.80 0.68 0.72 2000
weighted avg 0.84 0.85 0.83 2000
Confusion matrix
make_confusion_matrix(y_train, y_train_pred)
make_confusion_matrix(y_val, y_val_pred)
Observations¶
- Recall for this model is very low, with training and validation both in the mid to upper 40's
- Let's add momentum and see if it helps.
Neural Network (64,32,16,8) with SGD + Momentum (0.9)¶
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)
model_name = "Neural Network (64,32,16,8) with SGD + Momentum (0.9)"
#Initializing the neural network
sgd_momentum_model = Sequential()
sgd_momentum_model.add(Dense(64, activation='relu', input_dim = X_train.shape[1]))
sgd_momentum_model.add(Dense(32, activation='relu'))
sgd_momentum_model.add(Dense(16, activation='relu'))
sgd_momentum_model.add(Dense(8, activation='relu'))
sgd_momentum_model.add(Dense(1, activation = 'sigmoid'))
# use SGD as the optimizer.
optimizer = tf.keras.optimizers.SGD(momentum=0.9)
# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
check if this is the correct metric¶
## compile the model with binary cross entropy as loss function and recall as the metric.
sgd_momentum_model.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
sgd_momentum_model.summary()
Model: "sequential"
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāā ā Layer (type) ā Output Shape ā Param # ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā dense (Dense) ā (None, 64) ā 768 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_1 (Dense) ā (None, 32) ā 2,080 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_2 (Dense) ā (None, 16) ā 528 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_3 (Dense) ā (None, 8) ā 136 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_4 (Dense) ā (None, 1) ā 9 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāā
Total params: 3,521 (13.75 KB)
Trainable params: 3,521 (13.75 KB)
Non-trainable params: 0 (0.00 B)
check batch size and epochs¶
# Fitting the ANN
sgd_momentum_history = sgd_momentum_model.fit(
X_train, y_train,
batch_size=batch_size,
epochs=epochs,
validation_data=(X_val,y_val),
verbose=1
)
Epoch 1/50 94/94 āāāāāāāāāāāāāāāāāāāā 1s 3ms/step - loss: 0.5618 - recall: 0.0316 - val_loss: 0.4647 - val_recall: 0.0000e+00 Epoch 2/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4473 - recall: 0.0000e+00 - val_loss: 0.4293 - val_recall: 0.0025 Epoch 3/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4224 - recall: 0.0103 - val_loss: 0.4212 - val_recall: 0.1744 Epoch 4/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4119 - recall: 0.1734 - val_loss: 0.4120 - val_recall: 0.2604 Epoch 5/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4007 - recall: 0.2639 - val_loss: 0.3974 - val_recall: 0.3440 Epoch 6/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3850 - recall: 0.3386 - val_loss: 0.3800 - val_recall: 0.4472 Epoch 7/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3660 - recall: 0.4105 - val_loss: 0.3640 - val_recall: 0.4373 Epoch 8/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3530 - recall: 0.4427 - val_loss: 0.3569 - val_recall: 0.4349 Epoch 9/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3451 - recall: 0.4585 - val_loss: 0.3548 - val_recall: 0.4079 Epoch 10/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3404 - recall: 0.4674 - val_loss: 0.3524 - val_recall: 0.4201 Epoch 11/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3361 - recall: 0.4759 - val_loss: 0.3528 - val_recall: 0.4103 Epoch 12/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3331 - recall: 0.4810 - val_loss: 0.3531 - val_recall: 0.4029 Epoch 13/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3307 - recall: 0.4832 - val_loss: 0.3551 - val_recall: 0.3956 Epoch 14/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3289 - recall: 0.4916 - val_loss: 0.3552 - val_recall: 0.3907 Epoch 15/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3270 - recall: 0.4902 - val_loss: 0.3554 - val_recall: 0.4029 Epoch 16/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3254 - recall: 0.4926 - val_loss: 0.3546 - val_recall: 0.4054 Epoch 17/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3236 - recall: 0.4965 - val_loss: 0.3552 - val_recall: 0.3956 Epoch 18/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3223 - recall: 0.4980 - val_loss: 0.3560 - val_recall: 0.3956 Epoch 19/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3206 - recall: 0.4992 - val_loss: 0.3574 - val_recall: 0.3956 Epoch 20/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3192 - recall: 0.5048 - val_loss: 0.3589 - val_recall: 0.3980 Epoch 21/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3185 - recall: 0.5071 - val_loss: 0.3601 - val_recall: 0.3857 Epoch 22/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3175 - recall: 0.5069 - val_loss: 0.3610 - val_recall: 0.3857 Epoch 23/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3162 - recall: 0.5178 - val_loss: 0.3624 - val_recall: 0.3784 Epoch 24/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3153 - recall: 0.5122 - val_loss: 0.3629 - val_recall: 0.3808 Epoch 25/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3143 - recall: 0.5176 - val_loss: 0.3639 - val_recall: 0.3784 Epoch 26/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3135 - recall: 0.5193 - val_loss: 0.3645 - val_recall: 0.3808 Epoch 27/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3125 - recall: 0.5213 - val_loss: 0.3648 - val_recall: 0.3857 Epoch 28/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3115 - recall: 0.5217 - val_loss: 0.3666 - val_recall: 0.3833 Epoch 29/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3107 - recall: 0.5240 - val_loss: 0.3678 - val_recall: 0.3907 Epoch 30/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3100 - recall: 0.5184 - val_loss: 0.3682 - val_recall: 0.3907 Epoch 31/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3089 - recall: 0.5236 - val_loss: 0.3694 - val_recall: 0.3882 Epoch 32/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3078 - recall: 0.5206 - val_loss: 0.3703 - val_recall: 0.3882 Epoch 33/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3074 - recall: 0.5189 - val_loss: 0.3708 - val_recall: 0.3882 Epoch 34/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3061 - recall: 0.5249 - val_loss: 0.3720 - val_recall: 0.3907 Epoch 35/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3053 - recall: 0.5271 - val_loss: 0.3723 - val_recall: 0.3907 Epoch 36/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3043 - recall: 0.5308 - val_loss: 0.3731 - val_recall: 0.3907 Epoch 37/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3033 - recall: 0.5249 - val_loss: 0.3737 - val_recall: 0.3956 Epoch 38/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3027 - recall: 0.5345 - val_loss: 0.3760 - val_recall: 0.3882 Epoch 39/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3020 - recall: 0.5361 - val_loss: 0.3762 - val_recall: 0.3931 Epoch 40/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3010 - recall: 0.5354 - val_loss: 0.3779 - val_recall: 0.3907 Epoch 41/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3002 - recall: 0.5437 - val_loss: 0.3794 - val_recall: 0.3931 Epoch 42/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2995 - recall: 0.5389 - val_loss: 0.3787 - val_recall: 0.3882 Epoch 43/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2986 - recall: 0.5424 - val_loss: 0.3793 - val_recall: 0.3907 Epoch 44/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2975 - recall: 0.5476 - val_loss: 0.3796 - val_recall: 0.3980 Epoch 45/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2965 - recall: 0.5561 - val_loss: 0.3812 - val_recall: 0.4005 Epoch 46/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2959 - recall: 0.5489 - val_loss: 0.3820 - val_recall: 0.4029 Epoch 47/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2948 - recall: 0.5539 - val_loss: 0.3833 - val_recall: 0.4079 Epoch 48/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2938 - recall: 0.5508 - val_loss: 0.3846 - val_recall: 0.4029 Epoch 49/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2928 - recall: 0.5533 - val_loss: 0.3855 - val_recall: 0.4054 Epoch 50/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2922 - recall: 0.5543 - val_loss: 0.3862 - val_recall: 0.4054
Loss function
#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(loss=sgd_momentum_history.history['loss'],val_loss=sgd_history.history['val_loss'])
- The loss in the validation and train sets is not close but shows little noise
Recall
#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(sgd_momentum_history.history['recall'],sgd_history.history['val_recall'])
#Predicting the results using best as a threshold
y_train_pred = sgd_momentum_model.predict(X_train)
y_train_pred = (y_train_pred > 0.5)
y_train_pred
188/188 āāāāāāāāāāāāāāāāāāāā 0s 716us/step
array([[False],
[False],
[ True],
...,
[ True],
[False],
[ True]])
#Predicting the results using best as a threshold
y_val_pred = sgd_momentum_model.predict(X_val)
y_val_pred = (y_val_pred > 0.5)
y_val_pred
63/63 āāāāāāāāāāāāāāāāāāāā 0s 543us/step
array([[False],
[False],
[ True],
...,
[False],
[False],
[False]])
#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train, y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val, y_val_pred)
print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])
recall 0.518397 Name: Neural Network (64,32,16,8) with SGD + Momentum (0.9), dtype: float64 recall 0.405405 Name: Neural Network (64,32,16,8) with SGD + Momentum (0.9), dtype: float64
Classification report
#classification report
print(classification_report(y_train, y_train_pred))
precision recall f1-score support
0 0.89 0.98 0.93 4777
1 0.86 0.52 0.65 1223
accuracy 0.89 6000
macro avg 0.88 0.75 0.79 6000
weighted avg 0.88 0.89 0.87 6000
#classification report
print(classification_report(y_val, y_val_pred))
precision recall f1-score support
0 0.86 0.97 0.91 1593
1 0.77 0.41 0.53 407
accuracy 0.85 2000
macro avg 0.82 0.69 0.72 2000
weighted avg 0.85 0.85 0.84 2000
Confusion matrix
make_confusion_matrix(y_train, y_train_pred)
make_confusion_matrix(y_val, y_val_pred)
Observations¶
- Momentum improved recall performance on the traing set, but no improvemnt in validation.
- Let's try Adam this time.
Model Performance Improvement¶
Neural Network (64,32,16,8) with Adam Optimizer¶
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)
model_name = "Neural Network (64,32,16,8) with Adam"
#Initializing the neural network
adam_model = Sequential()
adam_model.add(Dense(64,activation='relu',input_dim = X_train.shape[1]))
adam_model.add(Dense(32,activation='relu'))
adam_model.add(Dense(16,activation='relu'))
adam_model.add(Dense(8,activation='relu'))
adam_model.add(Dense(1, activation = 'sigmoid'))
#use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam()
# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
# compile the model with binary cross entropy as loss function and recall as the metric
adam_model.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
adam_model.summary()
Model: "sequential"
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāā ā Layer (type) ā Output Shape ā Param # ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā dense (Dense) ā (None, 64) ā 768 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_1 (Dense) ā (None, 32) ā 2,080 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_2 (Dense) ā (None, 16) ā 528 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_3 (Dense) ā (None, 8) ā 136 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_4 (Dense) ā (None, 1) ā 9 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāā
Total params: 3,521 (13.75 KB)
Trainable params: 3,521 (13.75 KB)
Non-trainable params: 0 (0.00 B)
Loss function
#Fitting the ANN
adam_history = adam_model.fit(
X_train,y_train,
batch_size=batch_size,
epochs=epochs,
validation_data=(X_val,y_val),
verbose=1
)
Epoch 1/50 94/94 āāāāāāāāāāāāāāāāāāāā 1s 3ms/step - loss: 0.5593 - recall: 0.0227 - val_loss: 0.4319 - val_recall: 0.0074 Epoch 2/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4249 - recall: 0.0466 - val_loss: 0.4178 - val_recall: 0.1695 Epoch 3/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4067 - recall: 0.1933 - val_loss: 0.3960 - val_recall: 0.3243 Epoch 4/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3789 - recall: 0.3813 - val_loss: 0.3712 - val_recall: 0.4152 Epoch 5/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3547 - recall: 0.4507 - val_loss: 0.3596 - val_recall: 0.4423 Epoch 6/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3428 - recall: 0.4688 - val_loss: 0.3554 - val_recall: 0.4570 Epoch 7/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3355 - recall: 0.4841 - val_loss: 0.3535 - val_recall: 0.4595 Epoch 8/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3304 - recall: 0.4894 - val_loss: 0.3531 - val_recall: 0.4472 Epoch 9/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3260 - recall: 0.5006 - val_loss: 0.3532 - val_recall: 0.4349 Epoch 10/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3222 - recall: 0.5057 - val_loss: 0.3528 - val_recall: 0.4398 Epoch 11/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3185 - recall: 0.5157 - val_loss: 0.3526 - val_recall: 0.4423 Epoch 12/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3156 - recall: 0.5211 - val_loss: 0.3530 - val_recall: 0.4423 Epoch 13/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3126 - recall: 0.5262 - val_loss: 0.3538 - val_recall: 0.4496 Epoch 14/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3104 - recall: 0.5305 - val_loss: 0.3533 - val_recall: 0.4545 Epoch 15/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3085 - recall: 0.5389 - val_loss: 0.3538 - val_recall: 0.4521 Epoch 16/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3068 - recall: 0.5378 - val_loss: 0.3544 - val_recall: 0.4570 Epoch 17/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3048 - recall: 0.5410 - val_loss: 0.3553 - val_recall: 0.4545 Epoch 18/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3027 - recall: 0.5511 - val_loss: 0.3550 - val_recall: 0.4545 Epoch 19/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3003 - recall: 0.5499 - val_loss: 0.3557 - val_recall: 0.4570 Epoch 20/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2982 - recall: 0.5538 - val_loss: 0.3569 - val_recall: 0.4521 Epoch 21/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2963 - recall: 0.5538 - val_loss: 0.3578 - val_recall: 0.4447 Epoch 22/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2941 - recall: 0.5571 - val_loss: 0.3579 - val_recall: 0.4447 Epoch 23/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2924 - recall: 0.5578 - val_loss: 0.3580 - val_recall: 0.4472 Epoch 24/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2901 - recall: 0.5640 - val_loss: 0.3587 - val_recall: 0.4545 Epoch 25/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2879 - recall: 0.5724 - val_loss: 0.3588 - val_recall: 0.4545 Epoch 26/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2857 - recall: 0.5755 - val_loss: 0.3599 - val_recall: 0.4570 Epoch 27/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2843 - recall: 0.5718 - val_loss: 0.3611 - val_recall: 0.4595 Epoch 28/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2818 - recall: 0.5763 - val_loss: 0.3620 - val_recall: 0.4644 Epoch 29/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2797 - recall: 0.5771 - val_loss: 0.3635 - val_recall: 0.4619 Epoch 30/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2772 - recall: 0.5857 - val_loss: 0.3647 - val_recall: 0.4668 Epoch 31/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2747 - recall: 0.5888 - val_loss: 0.3663 - val_recall: 0.4717 Epoch 32/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2731 - recall: 0.5948 - val_loss: 0.3689 - val_recall: 0.4644 Epoch 33/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2714 - recall: 0.5935 - val_loss: 0.3717 - val_recall: 0.4717 Epoch 34/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2692 - recall: 0.5912 - val_loss: 0.3733 - val_recall: 0.4717 Epoch 35/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2673 - recall: 0.5944 - val_loss: 0.3760 - val_recall: 0.4668 Epoch 36/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2658 - recall: 0.6120 - val_loss: 0.3779 - val_recall: 0.4717 Epoch 37/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2630 - recall: 0.6123 - val_loss: 0.3796 - val_recall: 0.4742 Epoch 38/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2616 - recall: 0.6066 - val_loss: 0.3809 - val_recall: 0.4889 Epoch 39/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2595 - recall: 0.6215 - val_loss: 0.3831 - val_recall: 0.4742 Epoch 40/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2572 - recall: 0.6280 - val_loss: 0.3846 - val_recall: 0.4816 Epoch 41/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2556 - recall: 0.6305 - val_loss: 0.3867 - val_recall: 0.4791 Epoch 42/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2530 - recall: 0.6329 - val_loss: 0.3907 - val_recall: 0.4865 Epoch 43/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2513 - recall: 0.6369 - val_loss: 0.3963 - val_recall: 0.4914 Epoch 44/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2506 - recall: 0.6357 - val_loss: 0.3959 - val_recall: 0.4865 Epoch 45/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2478 - recall: 0.6400 - val_loss: 0.3985 - val_recall: 0.4791 Epoch 46/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2454 - recall: 0.6426 - val_loss: 0.4030 - val_recall: 0.4865 Epoch 47/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2441 - recall: 0.6492 - val_loss: 0.4049 - val_recall: 0.4742 Epoch 48/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2428 - recall: 0.6544 - val_loss: 0.4076 - val_recall: 0.4693 Epoch 49/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2411 - recall: 0.6521 - val_loss: 0.4110 - val_recall: 0.4742 Epoch 50/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2385 - recall: 0.6549 - val_loss: 0.4141 - val_recall: 0.4791
#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_history.history['loss'],adam_history.history['val_loss'])
- The loss in the validation and train sets is not close past epoch 5 and shows little noise
Recall
#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(adam_history.history['recall'],adam_history.history['val_recall'])
#Predicting the results using 0.5 as the threshold
y_train_pred = adam_model.predict(X_train)
y_train_pred = (y_train_pred > 0.5)
y_train_pred
188/188 āāāāāāāāāāāāāāāāāāāā 0s 744us/step
array([[False],
[False],
[ True],
...,
[ True],
[False],
[ True]])
#Predicting the results using 0.5 as the threshold
y_val_pred = adam_model.predict(X_val)
y_val_pred = (y_val_pred > 0.5)
y_val_pred
63/63 āāāāāāāāāāāāāāāāāāāā 0s 567us/step
array([[False],
[False],
[ True],
...,
[False],
[False],
[False]])
#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)
print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])
recall 0.651676 Name: Neural Network (64,32,16,8) with Adam, dtype: float64 recall 0.479115 Name: Neural Network (64,32,16,8) with Adam, dtype: float64
Classification report
#lassification report
print(classification_report(y_train,y_train_pred))
precision recall f1-score support
0 0.92 0.97 0.94 4777
1 0.85 0.65 0.74 1223
accuracy 0.91 6000
macro avg 0.88 0.81 0.84 6000
weighted avg 0.90 0.91 0.90 6000
#classification report
print(classification_report(y_val,y_val_pred))
precision recall f1-score support
0 0.88 0.94 0.91 1593
1 0.67 0.48 0.56 407
accuracy 0.85 2000
macro avg 0.77 0.71 0.73 2000
weighted avg 0.83 0.85 0.84 2000
Confusion matrix
#Calculating the confusion matrix
make_confusion_matrix(y_train, y_train_pred)
#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)
Observations¶
- Training recall improved, but validation got worse.
- Let's try increasing complexity with more layers to see what happens.
Neural Network (256,128,64,32,16,8) with Adam Optimizer¶
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)
model_name = "Neural Network (256,128,64,32,16,8) with Adam"
#Initializing the neural network
adam_model2 = Sequential()
adam_model2.add(Dense(256,activation='relu',input_dim = X_train.shape[1]))
adam_model2.add(Dense(128,activation='relu'))
adam_model2.add(Dense(64,activation='relu'))
adam_model2.add(Dense(32,activation='relu'))
adam_model2.add(Dense(16,activation='relu'))
adam_model2.add(Dense(8,activation='relu'))
adam_model2.add(Dense(1, activation = 'sigmoid'))
#use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam()
# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
# compile the model with binary cross entropy as loss function and recall as the metric
adam_model2.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
adam_model2.summary()
Model: "sequential"
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāā ā Layer (type) ā Output Shape ā Param # ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā dense (Dense) ā (None, 256) ā 3,072 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_1 (Dense) ā (None, 128) ā 32,896 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_2 (Dense) ā (None, 64) ā 8,256 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_3 (Dense) ā (None, 32) ā 2,080 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_4 (Dense) ā (None, 16) ā 528 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_5 (Dense) ā (None, 8) ā 136 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_6 (Dense) ā (None, 1) ā 9 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāā
Total params: 46,977 (183.50 KB)
Trainable params: 46,977 (183.50 KB)
Non-trainable params: 0 (0.00 B)
Loss function
#Fitting the ANN
adam_history2 = adam_model2.fit(
X_train,y_train,
batch_size=batch_size,
epochs=epochs,
validation_data=(X_val,y_val),
verbose=1
)
Epoch 1/50 94/94 āāāāāāāāāāāāāāāāāāāā 2s 4ms/step - loss: 0.5675 - recall: 0.1212 - val_loss: 0.4238 - val_recall: 0.2260 Epoch 2/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.4069 - recall: 0.2640 - val_loss: 0.3940 - val_recall: 0.3857 Epoch 3/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3706 - recall: 0.3913 - val_loss: 0.3704 - val_recall: 0.3661 Epoch 4/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3492 - recall: 0.4323 - val_loss: 0.3654 - val_recall: 0.3292 Epoch 5/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3398 - recall: 0.4311 - val_loss: 0.3629 - val_recall: 0.3292 Epoch 6/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3334 - recall: 0.4415 - val_loss: 0.3604 - val_recall: 0.3243 Epoch 7/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3285 - recall: 0.4303 - val_loss: 0.3608 - val_recall: 0.3096 Epoch 8/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3245 - recall: 0.4324 - val_loss: 0.3626 - val_recall: 0.2899 Epoch 9/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3203 - recall: 0.4293 - val_loss: 0.3617 - val_recall: 0.3243 Epoch 10/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3147 - recall: 0.4361 - val_loss: 0.3626 - val_recall: 0.3489 Epoch 11/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3100 - recall: 0.4435 - val_loss: 0.3635 - val_recall: 0.3317 Epoch 12/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3044 - recall: 0.4502 - val_loss: 0.3677 - val_recall: 0.3440 Epoch 13/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2969 - recall: 0.4680 - val_loss: 0.3692 - val_recall: 0.3587 Epoch 14/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2902 - recall: 0.4755 - val_loss: 0.3720 - val_recall: 0.3563 Epoch 15/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2831 - recall: 0.4944 - val_loss: 0.3791 - val_recall: 0.3563 Epoch 16/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2766 - recall: 0.4996 - val_loss: 0.3816 - val_recall: 0.3342 Epoch 17/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2713 - recall: 0.4892 - val_loss: 0.3921 - val_recall: 0.3489 Epoch 18/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2676 - recall: 0.5118 - val_loss: 0.3946 - val_recall: 0.3415 Epoch 19/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2607 - recall: 0.5508 - val_loss: 0.3996 - val_recall: 0.2998 Epoch 20/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2548 - recall: 0.5692 - val_loss: 0.4019 - val_recall: 0.3980 Epoch 21/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2522 - recall: 0.5747 - val_loss: 0.4142 - val_recall: 0.4570 Epoch 22/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2542 - recall: 0.6061 - val_loss: 0.4295 - val_recall: 0.4545 Epoch 23/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2489 - recall: 0.6085 - val_loss: 0.4745 - val_recall: 0.4791 Epoch 24/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2576 - recall: 0.5906 - val_loss: 0.4514 - val_recall: 0.4668 Epoch 25/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2497 - recall: 0.6210 - val_loss: 0.4547 - val_recall: 0.4693 Epoch 26/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2400 - recall: 0.6411 - val_loss: 0.4676 - val_recall: 0.4521 Epoch 27/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2371 - recall: 0.6449 - val_loss: 0.4509 - val_recall: 0.4275 Epoch 28/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2333 - recall: 0.6390 - val_loss: 0.4819 - val_recall: 0.4398 Epoch 29/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2187 - recall: 0.6514 - val_loss: 0.4898 - val_recall: 0.4079 Epoch 30/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2111 - recall: 0.6668 - val_loss: 0.4833 - val_recall: 0.4079 Epoch 31/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2036 - recall: 0.6893 - val_loss: 0.5300 - val_recall: 0.3415 Epoch 32/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1947 - recall: 0.6942 - val_loss: 0.5567 - val_recall: 0.4226 Epoch 33/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1857 - recall: 0.7270 - val_loss: 0.5749 - val_recall: 0.3636 Epoch 34/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1907 - recall: 0.7045 - val_loss: 0.5994 - val_recall: 0.3686 Epoch 35/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1944 - recall: 0.6973 - val_loss: 0.5827 - val_recall: 0.4398 Epoch 36/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1739 - recall: 0.7377 - val_loss: 0.5999 - val_recall: 0.4398 Epoch 37/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1705 - recall: 0.7551 - val_loss: 0.6237 - val_recall: 0.4226 Epoch 38/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1610 - recall: 0.7724 - val_loss: 0.6634 - val_recall: 0.4251 Epoch 39/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1581 - recall: 0.7681 - val_loss: 0.6665 - val_recall: 0.4742 Epoch 40/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1606 - recall: 0.7885 - val_loss: 0.6949 - val_recall: 0.5037 Epoch 41/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1610 - recall: 0.7880 - val_loss: 0.6720 - val_recall: 0.5012 Epoch 42/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1475 - recall: 0.7868 - val_loss: 0.7460 - val_recall: 0.5725 Epoch 43/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1567 - recall: 0.7951 - val_loss: 0.7438 - val_recall: 0.5405 Epoch 44/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1553 - recall: 0.7984 - val_loss: 0.7154 - val_recall: 0.5233 Epoch 45/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1333 - recall: 0.8104 - val_loss: 0.7832 - val_recall: 0.5627 Epoch 46/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1331 - recall: 0.8246 - val_loss: 0.8302 - val_recall: 0.5749 Epoch 47/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1460 - recall: 0.8182 - val_loss: 0.7945 - val_recall: 0.5749 Epoch 48/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1489 - recall: 0.8245 - val_loss: 0.8297 - val_recall: 0.5184 Epoch 49/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1236 - recall: 0.8519 - val_loss: 0.9069 - val_recall: 0.4742 Epoch 50/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1226 - recall: 0.8566 - val_loss: 0.9586 - val_recall: 0.4914
#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_history2.history['loss'],adam_history2.history['val_loss'])
- The loss in the validation and train sets diverges drastically and begins showing noise after epoch 20
#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(adam_history2.history['recall'],adam_history2.history['val_recall'])
#Predicting the results using 0.5 as the threshold
y_train_pred = adam_model2.predict(X_train)
y_train_pred = (y_train_pred > 0.5)
y_train_pred
188/188 āāāāāāāāāāāāāāāāāāāā 0s 913us/step
array([[False],
[ True],
[False],
...,
[ True],
[False],
[ True]])
#Predicting the results using 0.5 as the threshold
y_val_pred = adam_model2.predict(X_val)
y_val_pred = (y_val_pred > 0.5)
y_val_pred
63/63 āāāāāāāāāāāāāāāāāāāā 0s 668us/step
array([[False],
[False],
[ True],
...,
[False],
[False],
[False]])
#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)
print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])
recall 0.806214 Name: Neural Network (256,128,64,32,16,8) with Adam, dtype: float64 recall 0.4914 Name: Neural Network (256,128,64,32,16,8) with Adam, dtype: float64
Classification report
#lassification report
print(classification_report(y_train,y_train_pred))
precision recall f1-score support
0 0.95 0.98 0.97 4777
1 0.91 0.81 0.86 1223
accuracy 0.94 6000
macro avg 0.93 0.89 0.91 6000
weighted avg 0.94 0.94 0.94 6000
#classification report
print(classification_report(y_val,y_val_pred))
precision recall f1-score support
0 0.88 0.92 0.90 1593
1 0.61 0.49 0.54 407
accuracy 0.83 2000
macro avg 0.74 0.71 0.72 2000
weighted avg 0.82 0.83 0.83 2000
Confusion matrix
#Calculating the confusion matrix
make_confusion_matrix(y_train, y_train_pred)
#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)
Observations¶
- Training Recall improved quite a bit, but validation not so much.
- Training is probably overfitting the data.
- Let's see if adding a learning rate improves anything. We'll go with .001 to start
Neural Network (256,128,64,128,64,8) with Adam + LR (0.001)¶
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)
model_name = "Neural Network (256,128,64,128,64,8) with Adam + LR (0.001)"
#Initializing the neural network
adam_model3 = Sequential()
adam_model3.add(Dense(256,activation='relu',input_dim = X_train.shape[1]))
adam_model3.add(Dense(128,activation='relu'))
adam_model3.add(Dense(64,activation='relu'))
adam_model3.add(Dense(128,activation='relu'))
adam_model3.add(Dense(64,activation='relu'))
adam_model3.add(Dense(8,activation='relu'))
adam_model3.add(Dense(1, activation = 'sigmoid'))
#use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam(0.001)
# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
# compile the model with binary cross entropy as loss function and recall as the metric
adam_model3.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
adam_model.summary()
Model: "sequential"
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāā ā Layer (type) ā Output Shape ā Param # ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā dense (Dense) ā (None, 64) ā 768 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_1 (Dense) ā (None, 32) ā 2,080 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_2 (Dense) ā (None, 16) ā 528 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_3 (Dense) ā (None, 8) ā 136 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_4 (Dense) ā (None, 1) ā 9 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāā
Total params: 10,565 (41.27 KB)
Trainable params: 3,521 (13.75 KB)
Non-trainable params: 0 (0.00 B)
Optimizer params: 7,044 (27.52 KB)
Loss function
#Fitting the ANN
adam_history3 = adam_model3.fit(
X_train,y_train,
batch_size=batch_size,
epochs=epochs,
validation_data=(X_val,y_val),
verbose=1
)
Epoch 1/50 94/94 āāāāāāāāāāāāāāāāāāāā 2s 5ms/step - loss: 0.5314 - recall: 0.0664 - val_loss: 0.4201 - val_recall: 0.1892 Epoch 2/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3925 - recall: 0.3479 - val_loss: 0.3848 - val_recall: 0.3464 Epoch 3/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3557 - recall: 0.4013 - val_loss: 0.3611 - val_recall: 0.3882 Epoch 4/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3372 - recall: 0.4386 - val_loss: 0.3598 - val_recall: 0.3538 Epoch 5/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3301 - recall: 0.4380 - val_loss: 0.3578 - val_recall: 0.3735 Epoch 6/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3252 - recall: 0.4473 - val_loss: 0.3581 - val_recall: 0.3538 Epoch 7/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3180 - recall: 0.4527 - val_loss: 0.3597 - val_recall: 0.3636 Epoch 8/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3155 - recall: 0.4541 - val_loss: 0.3641 - val_recall: 0.3784 Epoch 9/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3091 - recall: 0.4625 - val_loss: 0.3668 - val_recall: 0.3342 Epoch 10/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3012 - recall: 0.4650 - val_loss: 0.3737 - val_recall: 0.3735 Epoch 11/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2966 - recall: 0.4745 - val_loss: 0.3770 - val_recall: 0.3735 Epoch 12/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2897 - recall: 0.4795 - val_loss: 0.3818 - val_recall: 0.3956 Epoch 13/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2851 - recall: 0.4917 - val_loss: 0.3992 - val_recall: 0.4005 Epoch 14/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2803 - recall: 0.5023 - val_loss: 0.3945 - val_recall: 0.3636 Epoch 15/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2724 - recall: 0.5077 - val_loss: 0.4352 - val_recall: 0.4791 Epoch 16/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2734 - recall: 0.5178 - val_loss: 0.4359 - val_recall: 0.4717 Epoch 17/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2688 - recall: 0.5218 - val_loss: 0.4138 - val_recall: 0.4545 Epoch 18/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2602 - recall: 0.5416 - val_loss: 0.4592 - val_recall: 0.4693 Epoch 19/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2544 - recall: 0.5503 - val_loss: 0.4677 - val_recall: 0.4447 Epoch 20/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2452 - recall: 0.5744 - val_loss: 0.5054 - val_recall: 0.4742 Epoch 21/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2462 - recall: 0.5372 - val_loss: 0.5107 - val_recall: 0.5135 Epoch 22/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2439 - recall: 0.5834 - val_loss: 0.5499 - val_recall: 0.5160 Epoch 23/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2273 - recall: 0.6164 - val_loss: 0.5675 - val_recall: 0.5184 Epoch 24/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2185 - recall: 0.6245 - val_loss: 0.6031 - val_recall: 0.5307 Epoch 25/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2087 - recall: 0.6313 - val_loss: 0.6230 - val_recall: 0.5283 Epoch 26/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2053 - recall: 0.6288 - val_loss: 0.6567 - val_recall: 0.5160 Epoch 27/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1957 - recall: 0.6672 - val_loss: 0.6784 - val_recall: 0.4963 Epoch 28/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1926 - recall: 0.6818 - val_loss: 0.6659 - val_recall: 0.4816 Epoch 29/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1961 - recall: 0.6715 - val_loss: 0.6566 - val_recall: 0.5111 Epoch 30/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1955 - recall: 0.7014 - val_loss: 0.6680 - val_recall: 0.4889 Epoch 31/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1947 - recall: 0.7060 - val_loss: 0.7147 - val_recall: 0.4816 Epoch 32/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1666 - recall: 0.7317 - val_loss: 0.7086 - val_recall: 0.4988 Epoch 33/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1648 - recall: 0.7386 - val_loss: 0.7528 - val_recall: 0.4644 Epoch 34/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1661 - recall: 0.7511 - val_loss: 0.7301 - val_recall: 0.4496 Epoch 35/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1459 - recall: 0.7724 - val_loss: 0.7509 - val_recall: 0.3882 Epoch 36/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1518 - recall: 0.7758 - val_loss: 0.6955 - val_recall: 0.4300 Epoch 37/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1515 - recall: 0.7655 - val_loss: 0.7487 - val_recall: 0.4324 Epoch 38/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1509 - recall: 0.7836 - val_loss: 0.7408 - val_recall: 0.4128 Epoch 39/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1314 - recall: 0.7919 - val_loss: 0.7819 - val_recall: 0.4275 Epoch 40/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1308 - recall: 0.8074 - val_loss: 0.9014 - val_recall: 0.4251 Epoch 41/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1220 - recall: 0.8158 - val_loss: 0.8531 - val_recall: 0.3661 Epoch 42/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1145 - recall: 0.8257 - val_loss: 0.8731 - val_recall: 0.3661 Epoch 43/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.0938 - recall: 0.8484 - val_loss: 1.0320 - val_recall: 0.3710 Epoch 44/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.0990 - recall: 0.8731 - val_loss: 1.1081 - val_recall: 0.3833 Epoch 45/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1201 - recall: 0.8200 - val_loss: 1.0196 - val_recall: 0.4251 Epoch 46/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1155 - recall: 0.8386 - val_loss: 0.9493 - val_recall: 0.4423 Epoch 47/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1038 - recall: 0.8656 - val_loss: 0.9677 - val_recall: 0.4742 Epoch 48/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.0958 - recall: 0.8803 - val_loss: 0.9661 - val_recall: 0.4693 Epoch 49/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.0862 - recall: 0.8984 - val_loss: 1.0386 - val_recall: 0.4349 Epoch 50/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.0950 - recall: 0.8892 - val_loss: 0.9834 - val_recall: 0.4570
#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_history3.history['loss'],adam_history3.history['val_loss'])
- The loss in the validation and train sets diverges drastically and begins showing noise after epoch 12
#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(adam_history3.history['recall'],adam_history3.history['val_recall'])
#Predicting the results using 0.5 as the threshold
y_train_pred = adam_model3.predict(X_train)
y_train_pred = (y_train_pred > 0.5)
y_train_pred
188/188 āāāāāāāāāāāāāāāāāāāā 0s 934us/step
array([[False],
[False],
[False],
...,
[ True],
[False],
[ True]])
#Predicting the results using 0.5 as the threshold
y_val_pred = adam_model3.predict(X_val)
y_val_pred = (y_val_pred > 0.5)
y_val_pred
63/63 āāāāāāāāāāāāāāāāāāāā 0s 629us/step
array([[False],
[False],
[False],
...,
[False],
[False],
[False]])
#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)
print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])
recall 0.876533 Name: Neural Network (256,128,64,128,64,8) with Adam + LR (0.001), dtype: float64 recall 0.457002 Name: Neural Network (256,128,64,128,64,8) with Adam + LR (0.001), dtype: float64
Classification report
#lassification report
print(classification_report(y_train,y_train_pred))
precision recall f1-score support
0 0.97 0.97 0.97 4777
1 0.87 0.88 0.87 1223
accuracy 0.95 6000
macro avg 0.92 0.92 0.92 6000
weighted avg 0.95 0.95 0.95 6000
#classification report
print(classification_report(y_val,y_val_pred))
precision recall f1-score support
0 0.87 0.89 0.88 1593
1 0.52 0.46 0.49 407
accuracy 0.80 2000
macro avg 0.69 0.67 0.68 2000
weighted avg 0.79 0.80 0.80 2000
Confusion matrix
#Calculating the confusion matrix
make_confusion_matrix(y_train, y_train_pred)
#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)
Observations¶
- About the same resuts as previous model, with high recall on training data but low on validation.
- Lets resimplify the model and add Dropout.
Neural Network (64,32,16,8) with Adam & Dropout (0.4, 0.3, 0.2, 0.1)¶
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)
model_name = "Neural Network (64,32,16,8) with Adam & Dropout (0.4, 0.3, 0.2, 0.1)"
#Initializing the neural network
adam_dropout_model = Sequential()
adam_dropout_model.add(Dense(64,activation='relu',input_dim = X_train.shape[1]))
adam_dropout_model.add(Dropout(0.4))
adam_dropout_model.add(Dense(32,activation='relu'))
adam_dropout_model.add(Dropout(0.3))
adam_dropout_model.add(Dense(16,activation='relu'))
adam_dropout_model.add(Dropout(0.2))
adam_dropout_model.add(Dense(8,activation='relu'))
adam_dropout_model.add(Dropout(0.1))
adam_dropout_model.add(Dense(1, activation = 'sigmoid'))
#Cuse Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam()
# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
## compile the model with binary cross entropy as loss function and recall as the metric.
adam_dropout_model.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
# Summary of the model
adam_dropout_model.summary()
Model: "sequential"
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāā ā Layer (type) ā Output Shape ā Param # ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā dense (Dense) ā (None, 64) ā 768 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout (Dropout) ā (None, 64) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_1 (Dense) ā (None, 32) ā 2,080 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_1 (Dropout) ā (None, 32) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_2 (Dense) ā (None, 16) ā 528 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_2 (Dropout) ā (None, 16) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_3 (Dense) ā (None, 8) ā 136 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_3 (Dropout) ā (None, 8) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_4 (Dense) ā (None, 1) ā 9 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāā
Total params: 3,521 (13.75 KB)
Trainable params: 3,521 (13.75 KB)
Non-trainable params: 0 (0.00 B)
Loss function
#Fitting the ANN
adam_dropout_history = adam_dropout_model.fit(
X_train,y_train,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data=(X_val,y_val)
)
Epoch 1/50 94/94 āāāāāāāāāāāāāāāāāāāā 2s 4ms/step - loss: 0.6397 - recall: 0.1852 - val_loss: 0.4683 - val_recall: 0.0000e+00 Epoch 2/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4878 - recall: 0.0000e+00 - val_loss: 0.4347 - val_recall: 0.0000e+00 Epoch 3/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4600 - recall: 1.3815e-04 - val_loss: 0.4331 - val_recall: 0.0000e+00 Epoch 4/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4593 - recall: 0.0059 - val_loss: 0.4305 - val_recall: 0.0000e+00 Epoch 5/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4513 - recall: 0.0253 - val_loss: 0.4259 - val_recall: 0.1204 Epoch 6/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4426 - recall: 0.1141 - val_loss: 0.4191 - val_recall: 0.2703 Epoch 7/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4381 - recall: 0.1852 - val_loss: 0.4107 - val_recall: 0.3022 Epoch 8/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4196 - recall: 0.2449 - val_loss: 0.4028 - val_recall: 0.3194 Epoch 9/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4148 - recall: 0.2763 - val_loss: 0.3937 - val_recall: 0.3612 Epoch 10/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4140 - recall: 0.2679 - val_loss: 0.3880 - val_recall: 0.4152 Epoch 11/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4043 - recall: 0.3106 - val_loss: 0.3813 - val_recall: 0.3563 Epoch 12/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3970 - recall: 0.2986 - val_loss: 0.3747 - val_recall: 0.3857 Epoch 13/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3954 - recall: 0.3161 - val_loss: 0.3727 - val_recall: 0.3661 Epoch 14/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3992 - recall: 0.3361 - val_loss: 0.3690 - val_recall: 0.3710 Epoch 15/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3845 - recall: 0.3516 - val_loss: 0.3641 - val_recall: 0.3882 Epoch 16/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3795 - recall: 0.3776 - val_loss: 0.3624 - val_recall: 0.4103 Epoch 17/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3846 - recall: 0.3780 - val_loss: 0.3614 - val_recall: 0.3980 Epoch 18/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3769 - recall: 0.3654 - val_loss: 0.3578 - val_recall: 0.4054 Epoch 19/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3695 - recall: 0.3802 - val_loss: 0.3575 - val_recall: 0.3833 Epoch 20/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3715 - recall: 0.3755 - val_loss: 0.3577 - val_recall: 0.3956 Epoch 21/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3688 - recall: 0.3859 - val_loss: 0.3543 - val_recall: 0.4251 Epoch 22/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3633 - recall: 0.4082 - val_loss: 0.3550 - val_recall: 0.4251 Epoch 23/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3644 - recall: 0.4186 - val_loss: 0.3537 - val_recall: 0.4668 Epoch 24/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3709 - recall: 0.4148 - val_loss: 0.3529 - val_recall: 0.4324 Epoch 25/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3649 - recall: 0.4213 - val_loss: 0.3520 - val_recall: 0.4668 Epoch 26/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3671 - recall: 0.4268 - val_loss: 0.3524 - val_recall: 0.4472 Epoch 27/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3645 - recall: 0.4198 - val_loss: 0.3511 - val_recall: 0.4496 Epoch 28/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3627 - recall: 0.4296 - val_loss: 0.3528 - val_recall: 0.4767 Epoch 29/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3538 - recall: 0.4636 - val_loss: 0.3524 - val_recall: 0.4693 Epoch 30/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3610 - recall: 0.4488 - val_loss: 0.3510 - val_recall: 0.4619 Epoch 31/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3580 - recall: 0.4316 - val_loss: 0.3511 - val_recall: 0.4619 Epoch 32/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3573 - recall: 0.4417 - val_loss: 0.3534 - val_recall: 0.4496 Epoch 33/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3617 - recall: 0.4456 - val_loss: 0.3525 - val_recall: 0.4570 Epoch 34/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3525 - recall: 0.4609 - val_loss: 0.3519 - val_recall: 0.4152 Epoch 35/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3525 - recall: 0.4288 - val_loss: 0.3517 - val_recall: 0.4644 Epoch 36/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3606 - recall: 0.4368 - val_loss: 0.3526 - val_recall: 0.4300 Epoch 37/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3591 - recall: 0.4310 - val_loss: 0.3517 - val_recall: 0.4545 Epoch 38/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3555 - recall: 0.4570 - val_loss: 0.3509 - val_recall: 0.4423 Epoch 39/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3582 - recall: 0.4433 - val_loss: 0.3509 - val_recall: 0.4595 Epoch 40/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3497 - recall: 0.4538 - val_loss: 0.3505 - val_recall: 0.4816 Epoch 41/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3569 - recall: 0.4594 - val_loss: 0.3491 - val_recall: 0.4889 Epoch 42/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3532 - recall: 0.4505 - val_loss: 0.3491 - val_recall: 0.4840 Epoch 43/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3480 - recall: 0.4535 - val_loss: 0.3519 - val_recall: 0.4619 Epoch 44/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3506 - recall: 0.4612 - val_loss: 0.3510 - val_recall: 0.4742 Epoch 45/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3542 - recall: 0.4575 - val_loss: 0.3516 - val_recall: 0.4717 Epoch 46/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3430 - recall: 0.4616 - val_loss: 0.3512 - val_recall: 0.4619 Epoch 47/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3541 - recall: 0.4611 - val_loss: 0.3513 - val_recall: 0.4619 Epoch 48/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3543 - recall: 0.4541 - val_loss: 0.3511 - val_recall: 0.4742 Epoch 49/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3468 - recall: 0.4609 - val_loss: 0.3502 - val_recall: 0.4693 Epoch 50/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3475 - recall: 0.4463 - val_loss: 0.3512 - val_recall: 0.4742
#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_dropout_history.history['loss'],adam_dropout_history.history['val_loss'])
- The loss in the validation and train sets converges again around epoch 26, but traing set shows a bit of noise due to dropout
#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(adam_dropout_history.history['recall'],adam_dropout_history.history['val_recall'])
#Predicting the results using best as a threshold
y_train_pred = adam_dropout_model.predict(X_train)
y_train_pred = (y_train_pred > 0.5)
y_train_pred
188/188 āāāāāāāāāāāāāāāāāāāā 0s 727us/step
array([[False],
[False],
[False],
...,
[ True],
[False],
[ True]])
#Predicting the results using 0.5 as the threshold.
y_val_pred = adam_dropout_model.predict(X_val)
y_val_pred = (y_val_pred > 0.5)
y_val_pred
63/63 āāāāāāāāāāāāāāāāāāāā 0s 568us/step
array([[False],
[False],
[False],
...,
[False],
[False],
[False]])
#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)
print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])
recall 0.510221 Name: Neural Network (64,32,16,8) with Adam & Dropout (0.4, 0.3, 0.2, 0.1), dtype: float64 recall 0.474201 Name: Neural Network (64,32,16,8) with Adam & Dropout (0.4, 0.3, 0.2, 0.1), dtype: float64
Classification report
#classification report
print(classification_report(y_train,y_train_pred))
precision recall f1-score support
0 0.88 0.96 0.92 4777
1 0.78 0.51 0.62 1223
accuracy 0.87 6000
macro avg 0.83 0.74 0.77 6000
weighted avg 0.86 0.87 0.86 6000
#classification report
print(classification_report(y_val,y_val_pred))
precision recall f1-score support
0 0.88 0.96 0.91 1593
1 0.73 0.47 0.58 407
accuracy 0.86 2000
macro avg 0.80 0.71 0.74 2000
weighted avg 0.85 0.86 0.85 2000
Confusion matrix
#Calculating the confusion matrix
make_confusion_matrix(y_train, y_train_pred)
#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)
Observations¶
- Terrible recall performance, so lets simplify and see.
- Accuracy and Precision seem to have improved, but we're not optimizing for those.
Neural Network (32,16,8) with Adam & Dropout (0.4, 0.3, 0.2)¶
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)
model_name = "Neural Network (32,16,8) with Adam & Dropout (0.4, 0.3, 0.2)"
#Initializing the neural network
adam_dropout_model2 = Sequential()
adam_dropout_model2.add(Dense(32,activation='relu',input_dim = X_train.shape[1]))
adam_dropout_model2.add(Dropout(0.4))
adam_dropout_model2.add(Dense(16,activation='relu'))
adam_dropout_model2.add(Dropout(0.3))
adam_dropout_model2.add(Dense(8,activation='relu'))
adam_dropout_model2.add(Dropout(0.2))
adam_dropout_model2.add(Dense(1, activation = 'sigmoid'))
# use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam()
# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
## compile the model with binary cross entropy as loss function and recall as the metric.
adam_dropout_model2.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
# Summary of the model
adam_dropout_model2.summary()
Model: "sequential"
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāā ā Layer (type) ā Output Shape ā Param # ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā dense (Dense) ā (None, 32) ā 384 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout (Dropout) ā (None, 32) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_1 (Dense) ā (None, 16) ā 528 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_1 (Dropout) ā (None, 16) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_2 (Dense) ā (None, 8) ā 136 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_2 (Dropout) ā (None, 8) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_3 (Dense) ā (None, 1) ā 9 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāā
Total params: 1,057 (4.13 KB)
Trainable params: 1,057 (4.13 KB)
Non-trainable params: 0 (0.00 B)
Loss function
#Fitting the ANN with batch_size = 32 and 100 epochs
adam_dropout_history2 = adam_dropout_model2.fit(
X_train,y_train,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data=(X_val,y_val)
)
Epoch 1/50 94/94 āāāāāāāāāāāāāāāāāāāā 1s 3ms/step - loss: 0.7092 - recall: 0.5137 - val_loss: 0.5181 - val_recall: 0.0000e+00 Epoch 2/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5431 - recall: 0.0227 - val_loss: 0.4635 - val_recall: 0.0000e+00 Epoch 3/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4995 - recall: 0.0240 - val_loss: 0.4447 - val_recall: 0.0074 Epoch 4/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4850 - recall: 0.0316 - val_loss: 0.4354 - val_recall: 0.0197 Epoch 5/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4705 - recall: 0.0701 - val_loss: 0.4313 - val_recall: 0.0221 Epoch 6/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4627 - recall: 0.0546 - val_loss: 0.4282 - val_recall: 0.0344 Epoch 7/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4590 - recall: 0.0693 - val_loss: 0.4256 - val_recall: 0.0516 Epoch 8/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4503 - recall: 0.0872 - val_loss: 0.4226 - val_recall: 0.0565 Epoch 9/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4655 - recall: 0.0922 - val_loss: 0.4205 - val_recall: 0.0639 Epoch 10/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4509 - recall: 0.1309 - val_loss: 0.4167 - val_recall: 0.0786 Epoch 11/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4451 - recall: 0.1626 - val_loss: 0.4113 - val_recall: 0.1400 Epoch 12/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4435 - recall: 0.1768 - val_loss: 0.4068 - val_recall: 0.1843 Epoch 13/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4318 - recall: 0.2181 - val_loss: 0.4014 - val_recall: 0.2604 Epoch 14/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4233 - recall: 0.2409 - val_loss: 0.3985 - val_recall: 0.2457 Epoch 15/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4240 - recall: 0.2342 - val_loss: 0.3937 - val_recall: 0.2703 Epoch 16/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4214 - recall: 0.2818 - val_loss: 0.3893 - val_recall: 0.2899 Epoch 17/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4188 - recall: 0.2649 - val_loss: 0.3861 - val_recall: 0.2678 Epoch 18/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4160 - recall: 0.2827 - val_loss: 0.3840 - val_recall: 0.2924 Epoch 19/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4144 - recall: 0.2873 - val_loss: 0.3806 - val_recall: 0.2948 Epoch 20/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4141 - recall: 0.2934 - val_loss: 0.3784 - val_recall: 0.3219 Epoch 21/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4094 - recall: 0.2991 - val_loss: 0.3764 - val_recall: 0.3415 Epoch 22/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4068 - recall: 0.3224 - val_loss: 0.3732 - val_recall: 0.3857 Epoch 23/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4111 - recall: 0.3443 - val_loss: 0.3707 - val_recall: 0.3563 Epoch 24/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4054 - recall: 0.3294 - val_loss: 0.3686 - val_recall: 0.3710 Epoch 25/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3997 - recall: 0.3212 - val_loss: 0.3686 - val_recall: 0.3538 Epoch 26/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4081 - recall: 0.3187 - val_loss: 0.3671 - val_recall: 0.3808 Epoch 27/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3942 - recall: 0.3328 - val_loss: 0.3672 - val_recall: 0.3661 Epoch 28/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3865 - recall: 0.3326 - val_loss: 0.3654 - val_recall: 0.3661 Epoch 29/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3918 - recall: 0.3440 - val_loss: 0.3641 - val_recall: 0.3636 Epoch 30/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3867 - recall: 0.3767 - val_loss: 0.3626 - val_recall: 0.3612 Epoch 31/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3874 - recall: 0.3537 - val_loss: 0.3619 - val_recall: 0.3636 Epoch 32/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3895 - recall: 0.3412 - val_loss: 0.3618 - val_recall: 0.3538 Epoch 33/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3877 - recall: 0.3362 - val_loss: 0.3596 - val_recall: 0.3636 Epoch 34/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3877 - recall: 0.3586 - val_loss: 0.3593 - val_recall: 0.3538 Epoch 35/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3841 - recall: 0.3439 - val_loss: 0.3596 - val_recall: 0.3489 Epoch 36/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3807 - recall: 0.3457 - val_loss: 0.3578 - val_recall: 0.3587 Epoch 37/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3790 - recall: 0.3639 - val_loss: 0.3572 - val_recall: 0.3710 Epoch 38/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3849 - recall: 0.3690 - val_loss: 0.3583 - val_recall: 0.3514 Epoch 39/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3770 - recall: 0.3875 - val_loss: 0.3556 - val_recall: 0.3686 Epoch 40/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3788 - recall: 0.3844 - val_loss: 0.3559 - val_recall: 0.3710 Epoch 41/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3819 - recall: 0.3742 - val_loss: 0.3573 - val_recall: 0.3636 Epoch 42/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3745 - recall: 0.3704 - val_loss: 0.3549 - val_recall: 0.3710 Epoch 43/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3722 - recall: 0.3902 - val_loss: 0.3545 - val_recall: 0.3735 Epoch 44/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3778 - recall: 0.3798 - val_loss: 0.3553 - val_recall: 0.3661 Epoch 45/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3728 - recall: 0.3964 - val_loss: 0.3553 - val_recall: 0.3563 Epoch 46/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3818 - recall: 0.3736 - val_loss: 0.3547 - val_recall: 0.3735 Epoch 47/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3732 - recall: 0.3956 - val_loss: 0.3535 - val_recall: 0.3980 Epoch 48/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3705 - recall: 0.4011 - val_loss: 0.3551 - val_recall: 0.3735 Epoch 49/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3699 - recall: 0.3685 - val_loss: 0.3544 - val_recall: 0.3833 Epoch 50/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3670 - recall: 0.3782 - val_loss: 0.3537 - val_recall: 0.3833
#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_dropout_history2.history['loss'],adam_dropout_history2.history['val_loss'])
- The loss in the validation and train are similar, though divergent, and some noise in both sets though much more in triang due to dropout
#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(adam_dropout_history2.history['recall'],adam_dropout_history2.history['val_recall'])
#Predicting the results using best as a threshold
y_train_pred = adam_dropout_model2.predict(X_train)
y_train_pred = (y_train_pred > 0.5)
y_train_pred
188/188 āāāāāāāāāāāāāāāāāāāā 0s 680us/step
array([[False],
[False],
[False],
...,
[ True],
[False],
[ True]])
#Predicting the results using 0.5 as the threshold.
y_val_pred = adam_dropout_model2.predict(X_val)
y_val_pred = (y_val_pred > 0.5)
y_val_pred
63/63 āāāāāāāāāāāāāāāāāāāā 0s 544us/step
array([[False],
[False],
[False],
...,
[False],
[False],
[False]])
#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)
print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])
recall 0.420278 Name: Neural Network (32,16,8) with Adam & Dropout (0.4, 0.3, 0.2), dtype: float64 recall 0.383292 Name: Neural Network (32,16,8) with Adam & Dropout (0.4, 0.3, 0.2), dtype: float64
Classification report
#classification report
print(classification_report(y_train,y_train_pred))
precision recall f1-score support
0 0.87 0.98 0.92 4777
1 0.81 0.42 0.55 1223
accuracy 0.86 6000
macro avg 0.84 0.70 0.74 6000
weighted avg 0.86 0.86 0.84 6000
#classification report
print(classification_report(y_val,y_val_pred))
precision recall f1-score support
0 0.86 0.97 0.91 1593
1 0.77 0.38 0.51 407
accuracy 0.85 2000
macro avg 0.82 0.68 0.71 2000
weighted avg 0.84 0.85 0.83 2000
Confusion matrix
#Calculating the confusion matrix
make_confusion_matrix(y_train, y_train_pred)
#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)
Observations¶
- Still pretty terrible performance on recall.
- Lets simplify again and raise dropout
Neural Network (64,32) with Adam & Dropout (0.5, 0.5)¶
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)
model_name = "Neural Network (64,32) with Adam & Dropout (0.5, 0.5)"
#Initializing the neural network
adam_dropout_model3 = Sequential()
adam_dropout_model3.add(Dense(64,activation='relu',input_dim = X_train.shape[1]))
adam_dropout_model3.add(Dropout(0.5))
adam_dropout_model3.add(Dense(32,activation='relu'))
adam_dropout_model3.add(Dropout(0.5))
adam_dropout_model3.add(Dense(1, activation = 'sigmoid'))
# use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam(0.01)
# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
## compile the model with binary cross entropy as loss function and recall as the metric.
adam_dropout_model3.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
# Summary of the model
adam_dropout_model3.summary()
Model: "sequential"
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāā ā Layer (type) ā Output Shape ā Param # ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā dense (Dense) ā (None, 64) ā 768 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout (Dropout) ā (None, 64) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_1 (Dense) ā (None, 32) ā 2,080 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_1 (Dropout) ā (None, 32) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_2 (Dense) ā (None, 1) ā 33 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāā
Total params: 2,881 (11.25 KB)
Trainable params: 2,881 (11.25 KB)
Non-trainable params: 0 (0.00 B)
Loss function
#Fitting the ANN with batch_size = 32 and 100 epochs
adam_dropout_history3 = adam_dropout_model3.fit(
X_train,y_train,
batch_size=32,
epochs=epochs,
verbose=1,
validation_data=(X_val,y_val)
)
Epoch 1/50 188/188 āāāāāāāāāāāāāāāāāāāā 1s 2ms/step - loss: 0.4951 - recall: 0.0881 - val_loss: 0.4170 - val_recall: 0.0737 Epoch 2/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4314 - recall: 0.1775 - val_loss: 0.3998 - val_recall: 0.1695 Epoch 3/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4235 - recall: 0.1973 - val_loss: 0.3823 - val_recall: 0.2162 Epoch 4/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4125 - recall: 0.2549 - val_loss: 0.3717 - val_recall: 0.3022 Epoch 5/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3973 - recall: 0.2854 - val_loss: 0.3701 - val_recall: 0.2678 Epoch 6/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3948 - recall: 0.3105 - val_loss: 0.3707 - val_recall: 0.2924 Epoch 7/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3886 - recall: 0.2783 - val_loss: 0.3664 - val_recall: 0.3391 Epoch 8/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3782 - recall: 0.3142 - val_loss: 0.3681 - val_recall: 0.3268 Epoch 9/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3789 - recall: 0.3132 - val_loss: 0.3612 - val_recall: 0.3145 Epoch 10/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3797 - recall: 0.3382 - val_loss: 0.3599 - val_recall: 0.3243 Epoch 11/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3762 - recall: 0.3512 - val_loss: 0.3661 - val_recall: 0.2752 Epoch 12/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3904 - recall: 0.3121 - val_loss: 0.3698 - val_recall: 0.3194 Epoch 13/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3696 - recall: 0.3453 - val_loss: 0.3655 - val_recall: 0.3268 Epoch 14/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3751 - recall: 0.3487 - val_loss: 0.3666 - val_recall: 0.2899 Epoch 15/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3663 - recall: 0.3359 - val_loss: 0.3636 - val_recall: 0.3096 Epoch 16/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3811 - recall: 0.3266 - val_loss: 0.3664 - val_recall: 0.2850 Epoch 17/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3779 - recall: 0.3278 - val_loss: 0.3656 - val_recall: 0.3292 Epoch 18/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3779 - recall: 0.3248 - val_loss: 0.3630 - val_recall: 0.3538 Epoch 19/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3777 - recall: 0.3231 - val_loss: 0.3649 - val_recall: 0.2875 Epoch 20/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3753 - recall: 0.3257 - val_loss: 0.3641 - val_recall: 0.3096 Epoch 21/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3719 - recall: 0.3760 - val_loss: 0.3647 - val_recall: 0.2973 Epoch 22/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3685 - recall: 0.3576 - val_loss: 0.3618 - val_recall: 0.3440 Epoch 23/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3772 - recall: 0.3307 - val_loss: 0.3645 - val_recall: 0.3391 Epoch 24/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3743 - recall: 0.3635 - val_loss: 0.3709 - val_recall: 0.2776 Epoch 25/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3676 - recall: 0.3310 - val_loss: 0.3627 - val_recall: 0.3170 Epoch 26/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3768 - recall: 0.3379 - val_loss: 0.3614 - val_recall: 0.3415 Epoch 27/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3776 - recall: 0.3282 - val_loss: 0.3610 - val_recall: 0.3292 Epoch 28/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3676 - recall: 0.3543 - val_loss: 0.3610 - val_recall: 0.3759 Epoch 29/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3717 - recall: 0.3546 - val_loss: 0.3621 - val_recall: 0.3538 Epoch 30/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3728 - recall: 0.3364 - val_loss: 0.3629 - val_recall: 0.3514 Epoch 31/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3662 - recall: 0.3522 - val_loss: 0.3628 - val_recall: 0.3612 Epoch 32/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3711 - recall: 0.3490 - val_loss: 0.3612 - val_recall: 0.3440 Epoch 33/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3718 - recall: 0.3476 - val_loss: 0.3629 - val_recall: 0.3612 Epoch 34/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3807 - recall: 0.3550 - val_loss: 0.3613 - val_recall: 0.3022 Epoch 35/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3637 - recall: 0.3595 - val_loss: 0.3635 - val_recall: 0.2998 Epoch 36/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3708 - recall: 0.3556 - val_loss: 0.3647 - val_recall: 0.3710 Epoch 37/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3635 - recall: 0.3807 - val_loss: 0.3591 - val_recall: 0.3808 Epoch 38/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3731 - recall: 0.3692 - val_loss: 0.3609 - val_recall: 0.3538 Epoch 39/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3653 - recall: 0.3576 - val_loss: 0.3652 - val_recall: 0.3464 Epoch 40/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3758 - recall: 0.3405 - val_loss: 0.3612 - val_recall: 0.3464 Epoch 41/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3652 - recall: 0.3507 - val_loss: 0.3660 - val_recall: 0.3489 Epoch 42/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3725 - recall: 0.3480 - val_loss: 0.3632 - val_recall: 0.3857 Epoch 43/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3792 - recall: 0.3132 - val_loss: 0.3689 - val_recall: 0.3170 Epoch 44/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3752 - recall: 0.3390 - val_loss: 0.3630 - val_recall: 0.3342 Epoch 45/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3596 - recall: 0.3699 - val_loss: 0.3604 - val_recall: 0.3464 Epoch 46/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3587 - recall: 0.3550 - val_loss: 0.3573 - val_recall: 0.4005 Epoch 47/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3633 - recall: 0.3827 - val_loss: 0.3618 - val_recall: 0.3538 Epoch 48/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3717 - recall: 0.3571 - val_loss: 0.3646 - val_recall: 0.3440 Epoch 49/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3664 - recall: 0.3352 - val_loss: 0.3766 - val_recall: 0.2678 Epoch 50/50 188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3732 - recall: 0.3182 - val_loss: 0.3646 - val_recall: 0.3047
#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_dropout_history3.history['loss'],adam_dropout_history3.history['val_loss'])
- The loss in the validation and train are similar, though divergent, and showing more noise in both sets
#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(adam_dropout_history3.history['recall'],adam_dropout_history3.history['val_recall'])
#Predicting the results using best as a threshold
y_train_pred = adam_dropout_model3.predict(X_train)
y_train_pred = (y_train_pred > 0.5)
y_train_pred
188/188 āāāāāāāāāāāāāāāāāāāā 0s 961us/step
array([[False],
[False],
[False],
...,
[ True],
[False],
[ True]])
#Predicting the results using 0.5 as the threshold.
y_val_pred = adam_dropout_model3.predict(X_val)
y_val_pred = (y_val_pred > 0.5)
y_val_pred
63/63 āāāāāāāāāāāāāāāāāāāā 0s 573us/step
array([[False],
[False],
[False],
...,
[False],
[False],
[False]])
#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)
print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])
recall 0.348324 Name: Neural Network (64,32) with Adam & Dropout (0.5, 0.5), dtype: float64 recall 0.304668 Name: Neural Network (64,32) with Adam & Dropout (0.5, 0.5), dtype: float64
Classification report
#classification report
print(classification_report(y_train,y_train_pred))
precision recall f1-score support
0 0.86 0.99 0.92 4777
1 0.91 0.35 0.50 1223
accuracy 0.86 6000
macro avg 0.88 0.67 0.71 6000
weighted avg 0.87 0.86 0.83 6000
#classification report
print(classification_report(y_val,y_val_pred))
precision recall f1-score support
0 0.85 0.98 0.91 1593
1 0.83 0.30 0.45 407
accuracy 0.85 2000
macro avg 0.84 0.64 0.68 2000
weighted avg 0.84 0.85 0.82 2000
Confusion matrix
#Calculating the confusion matrix
make_confusion_matrix(y_train, y_train_pred)
#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)
Observations¶
- This model is likely too simple to really do much.
- Terrible recall performance.
- Add some layers back infor next model.
- Try He normal to see how it affects a previous model.
Neural Network (64,32,16,8) with Adam, He Normal & Dropout (0.4, 0.3, 0.2, 0.1)¶
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)
model_name = "Neural Network (64,32,16,8) with Adam, He Normal & Dropout (0.4, 0.3, 0.2, 0.1)"
#Initializing the neural network
adam_he_dropout_model = Sequential()
adam_he_dropout_model.add(Dense(64,activation='relu', kernel_initializer="he_normal", input_dim = X_train.shape[1]))
adam_he_dropout_model.add(Dropout(0.4))
adam_he_dropout_model.add(Dense(32,activation='relu', kernel_initializer="he_normal"))
adam_he_dropout_model.add(Dropout(0.3))
adam_he_dropout_model.add(Dense(16,activation='relu', kernel_initializer="he_normal"))
adam_he_dropout_model.add(Dropout(0.2))
adam_he_dropout_model.add(Dense(8,activation='relu', kernel_initializer="he_normal"))
adam_he_dropout_model.add(Dropout(0.1))
adam_he_dropout_model.add(Dense(1, activation = 'sigmoid', kernel_initializer="he_normal"))
# use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam()
# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
## compile the model with binary cross entropy as loss function and recall as the metric.
adam_he_dropout_model.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
# Summary of the model
adam_he_dropout_model.summary()
Model: "sequential"
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāā ā Layer (type) ā Output Shape ā Param # ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā dense (Dense) ā (None, 64) ā 768 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout (Dropout) ā (None, 64) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_1 (Dense) ā (None, 32) ā 2,080 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_1 (Dropout) ā (None, 32) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_2 (Dense) ā (None, 16) ā 528 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_2 (Dropout) ā (None, 16) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_3 (Dense) ā (None, 8) ā 136 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_3 (Dropout) ā (None, 8) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_4 (Dense) ā (None, 1) ā 9 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāā
Total params: 3,521 (13.75 KB)
Trainable params: 3,521 (13.75 KB)
Non-trainable params: 0 (0.00 B)
Loss function
#Fitting the ANN
adam_he_dropout_history = adam_he_dropout_model.fit(
X_train,y_train,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data=(X_val,y_val)
)
Epoch 1/50 94/94 āāāāāāāāāāāāāāāāāāāā 2s 4ms/step - loss: 0.7711 - recall: 0.4888 - val_loss: 0.4644 - val_recall: 0.0000e+00 Epoch 2/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5203 - recall: 0.0572 - val_loss: 0.4496 - val_recall: 0.0000e+00 Epoch 3/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4879 - recall: 0.0408 - val_loss: 0.4407 - val_recall: 0.0000e+00 Epoch 4/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4721 - recall: 0.0616 - val_loss: 0.4360 - val_recall: 0.0000e+00 Epoch 5/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4680 - recall: 0.0574 - val_loss: 0.4321 - val_recall: 0.0000e+00 Epoch 6/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4641 - recall: 0.0563 - val_loss: 0.4275 - val_recall: 0.0025 Epoch 7/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4593 - recall: 0.0803 - val_loss: 0.4260 - val_recall: 0.0000e+00 Epoch 8/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4505 - recall: 0.0973 - val_loss: 0.4241 - val_recall: 0.0098 Epoch 9/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4448 - recall: 0.1195 - val_loss: 0.4209 - val_recall: 0.0172 Epoch 10/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4479 - recall: 0.1025 - val_loss: 0.4196 - val_recall: 0.0565 Epoch 11/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4404 - recall: 0.1329 - val_loss: 0.4137 - val_recall: 0.1376 Epoch 12/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4460 - recall: 0.1611 - val_loss: 0.4105 - val_recall: 0.1622 Epoch 13/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4384 - recall: 0.1617 - val_loss: 0.4075 - val_recall: 0.1867 Epoch 14/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4291 - recall: 0.1844 - val_loss: 0.4045 - val_recall: 0.1990 Epoch 15/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4241 - recall: 0.1818 - val_loss: 0.4021 - val_recall: 0.2088 Epoch 16/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4246 - recall: 0.2188 - val_loss: 0.3968 - val_recall: 0.2383 Epoch 17/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4352 - recall: 0.2378 - val_loss: 0.3954 - val_recall: 0.2334 Epoch 18/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4145 - recall: 0.2265 - val_loss: 0.3918 - val_recall: 0.2555 Epoch 19/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4244 - recall: 0.2518 - val_loss: 0.3903 - val_recall: 0.2383 Epoch 20/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4175 - recall: 0.2456 - val_loss: 0.3862 - val_recall: 0.2850 Epoch 21/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4088 - recall: 0.2808 - val_loss: 0.3846 - val_recall: 0.2654 Epoch 22/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4045 - recall: 0.2916 - val_loss: 0.3796 - val_recall: 0.3268 Epoch 23/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4059 - recall: 0.2760 - val_loss: 0.3784 - val_recall: 0.3120 Epoch 24/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4095 - recall: 0.2726 - val_loss: 0.3773 - val_recall: 0.3022 Epoch 25/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3934 - recall: 0.3055 - val_loss: 0.3730 - val_recall: 0.3194 Epoch 26/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3853 - recall: 0.3311 - val_loss: 0.3707 - val_recall: 0.3194 Epoch 27/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3954 - recall: 0.3212 - val_loss: 0.3707 - val_recall: 0.3268 Epoch 28/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3906 - recall: 0.3416 - val_loss: 0.3700 - val_recall: 0.3071 Epoch 29/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3878 - recall: 0.3264 - val_loss: 0.3670 - val_recall: 0.3268 Epoch 30/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3959 - recall: 0.3285 - val_loss: 0.3676 - val_recall: 0.3145 Epoch 31/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3835 - recall: 0.3339 - val_loss: 0.3638 - val_recall: 0.3686 Epoch 32/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3867 - recall: 0.3409 - val_loss: 0.3643 - val_recall: 0.3464 Epoch 33/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3856 - recall: 0.3361 - val_loss: 0.3639 - val_recall: 0.3464 Epoch 34/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3774 - recall: 0.3673 - val_loss: 0.3624 - val_recall: 0.3538 Epoch 35/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3793 - recall: 0.3680 - val_loss: 0.3606 - val_recall: 0.3661 Epoch 36/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3826 - recall: 0.3462 - val_loss: 0.3610 - val_recall: 0.3366 Epoch 37/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3730 - recall: 0.3876 - val_loss: 0.3600 - val_recall: 0.3612 Epoch 38/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3696 - recall: 0.3866 - val_loss: 0.3594 - val_recall: 0.3686 Epoch 39/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3741 - recall: 0.3809 - val_loss: 0.3620 - val_recall: 0.3022 Epoch 40/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3677 - recall: 0.3827 - val_loss: 0.3591 - val_recall: 0.3538 Epoch 41/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3694 - recall: 0.3876 - val_loss: 0.3597 - val_recall: 0.3415 Epoch 42/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3730 - recall: 0.3495 - val_loss: 0.3581 - val_recall: 0.3538 Epoch 43/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3736 - recall: 0.3668 - val_loss: 0.3587 - val_recall: 0.3391 Epoch 44/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3753 - recall: 0.3758 - val_loss: 0.3579 - val_recall: 0.3464 Epoch 45/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3690 - recall: 0.4118 - val_loss: 0.3587 - val_recall: 0.3464 Epoch 46/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3737 - recall: 0.3848 - val_loss: 0.3576 - val_recall: 0.3440 Epoch 47/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3650 - recall: 0.3883 - val_loss: 0.3575 - val_recall: 0.3440 Epoch 48/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3628 - recall: 0.4153 - val_loss: 0.3575 - val_recall: 0.3366 Epoch 49/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3566 - recall: 0.3842 - val_loss: 0.3563 - val_recall: 0.3612 Epoch 50/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3595 - recall: 0.4283 - val_loss: 0.3576 - val_recall: 0.3317
#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_he_dropout_history.history['loss'],adam_he_dropout_history.history['val_loss'])
- The loss in the validation and train are similar, though divergent, and some noise in both sets though much more in training due to dropout
#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(adam_he_dropout_history.history['recall'],adam_he_dropout_history.history['val_recall'])
#Predicting the results using best as a threshold
y_train_pred = adam_he_dropout_model.predict(X_train)
y_train_pred = (y_train_pred > 0.5)
y_train_pred
188/188 āāāāāāāāāāāāāāāāāāāā 0s 725us/step
array([[False],
[False],
[False],
...,
[ True],
[False],
[ True]])
#Predicting the results using 0.5 as the threshold.
y_val_pred = adam_he_dropout_model.predict(X_val)
y_val_pred = (y_val_pred > 0.5)
y_val_pred
63/63 āāāāāāāāāāāāāāāāāāāā 0s 541us/step
array([[False],
[False],
[False],
...,
[False],
[False],
[False]])
#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)
print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])
recall 0.376124 Name: Neural Network (64,32,16,8) with Adam, He Normal & Dropout (0.4, 0.3, 0.2, 0.1), dtype: float64 recall 0.331695 Name: Neural Network (64,32,16,8) with Adam, He Normal & Dropout (0.4, 0.3, 0.2, 0.1), dtype: float64
Classification report
#classification report
print(classification_report(y_train,y_train_pred))
precision recall f1-score support
0 0.86 0.99 0.92 4777
1 0.87 0.38 0.53 1223
accuracy 0.86 6000
macro avg 0.87 0.68 0.72 6000
weighted avg 0.86 0.86 0.84 6000
#classification report
print(classification_report(y_val,y_val_pred))
precision recall f1-score support
0 0.85 0.98 0.91 1593
1 0.78 0.33 0.47 407
accuracy 0.85 2000
macro avg 0.82 0.65 0.69 2000
weighted avg 0.84 0.85 0.82 2000
Confusion matrix
#Calculating the confusion matrix
make_confusion_matrix(y_train, y_train_pred)
#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)
Observations¶
- Definitely going the wrong way on these models.
- he normal does not seem to have helped.
- Recall is not acceptable.
- Resimplify and lets try some batch normalization.
Neural Network (64,32,16,8) with Adam & Batch Normalization¶
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)
model_name = "Neural Network (64,32,16,8) with Adam & Batch Normalization"
#Initializing the neural network
adam_batch_model = Sequential()
adam_batch_model.add(Dense(64,activation='relu',input_dim = X_train.shape[1]))
adam_batch_model.add(BatchNormalization())
adam_batch_model.add(Dense(32,activation='relu'))
adam_batch_model.add(BatchNormalization())
adam_batch_model.add(Dense(16,activation='relu'))
adam_batch_model.add(BatchNormalization())
adam_batch_model.add(Dense(8,activation='relu'))
adam_batch_model.add(BatchNormalization())
adam_batch_model.add(Dense(1, activation = 'sigmoid'))
# use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam()
# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
## compile the model with binary cross entropy as loss function and recall as the metric.
adam_batch_model.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
# Summary of the model
adam_batch_model.summary()
Model: "sequential"
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāā ā Layer (type) ā Output Shape ā Param # ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā dense (Dense) ā (None, 64) ā 768 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā batch_normalization ā (None, 64) ā 256 ā ā (BatchNormalization) ā ā ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_1 (Dense) ā (None, 32) ā 2,080 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā batch_normalization_1 ā (None, 32) ā 128 ā ā (BatchNormalization) ā ā ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_2 (Dense) ā (None, 16) ā 528 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā batch_normalization_2 ā (None, 16) ā 64 ā ā (BatchNormalization) ā ā ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_3 (Dense) ā (None, 8) ā 136 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā batch_normalization_3 ā (None, 8) ā 32 ā ā (BatchNormalization) ā ā ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_4 (Dense) ā (None, 1) ā 9 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāā
Total params: 4,001 (15.63 KB)
Trainable params: 3,761 (14.69 KB)
Non-trainable params: 240 (960.00 B)
Loss function
#Fitting the ANN with
adam_batch_history = adam_batch_model.fit(
X_train,y_train,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data=(X_val,y_val)
)
Epoch 1/50 94/94 āāāāāāāāāāāāāāāāāāāā 3s 5ms/step - loss: 0.8402 - recall: 0.6037 - val_loss: 0.5860 - val_recall: 0.2899 Epoch 2/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.5146 - recall: 0.5745 - val_loss: 0.4742 - val_recall: 0.2752 Epoch 3/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.4283 - recall: 0.5071 - val_loss: 0.4114 - val_recall: 0.3170 Epoch 4/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3754 - recall: 0.4960 - val_loss: 0.3840 - val_recall: 0.3735 Epoch 5/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3444 - recall: 0.4964 - val_loss: 0.3733 - val_recall: 0.3931 Epoch 6/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3271 - recall: 0.4989 - val_loss: 0.3716 - val_recall: 0.3931 Epoch 7/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3126 - recall: 0.5199 - val_loss: 0.3724 - val_recall: 0.3907 Epoch 8/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3025 - recall: 0.5286 - val_loss: 0.3734 - val_recall: 0.3931 Epoch 9/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2928 - recall: 0.5376 - val_loss: 0.3760 - val_recall: 0.4054 Epoch 10/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2840 - recall: 0.5670 - val_loss: 0.3806 - val_recall: 0.4201 Epoch 11/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2762 - recall: 0.5836 - val_loss: 0.3837 - val_recall: 0.4300 Epoch 12/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2686 - recall: 0.6001 - val_loss: 0.3880 - val_recall: 0.4275 Epoch 13/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2615 - recall: 0.6027 - val_loss: 0.3960 - val_recall: 0.4545 Epoch 14/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2536 - recall: 0.6203 - val_loss: 0.4032 - val_recall: 0.4423 Epoch 15/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2465 - recall: 0.6363 - val_loss: 0.4059 - val_recall: 0.4349 Epoch 16/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2376 - recall: 0.6507 - val_loss: 0.4157 - val_recall: 0.4668 Epoch 17/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2340 - recall: 0.6598 - val_loss: 0.4234 - val_recall: 0.4398 Epoch 18/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2266 - recall: 0.6708 - val_loss: 0.4382 - val_recall: 0.4595 Epoch 19/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2215 - recall: 0.6858 - val_loss: 0.4471 - val_recall: 0.4693 Epoch 20/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2165 - recall: 0.7009 - val_loss: 0.4567 - val_recall: 0.4644 Epoch 21/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2089 - recall: 0.6971 - val_loss: 0.4639 - val_recall: 0.4545 Epoch 22/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.2028 - recall: 0.7081 - val_loss: 0.4743 - val_recall: 0.4619 Epoch 23/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1990 - recall: 0.7232 - val_loss: 0.4882 - val_recall: 0.4521 Epoch 24/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1938 - recall: 0.7346 - val_loss: 0.4949 - val_recall: 0.4472 Epoch 25/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1842 - recall: 0.7461 - val_loss: 0.5075 - val_recall: 0.4324 Epoch 26/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1801 - recall: 0.7626 - val_loss: 0.5213 - val_recall: 0.4472 Epoch 27/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1769 - recall: 0.7582 - val_loss: 0.5318 - val_recall: 0.4398 Epoch 28/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1700 - recall: 0.7787 - val_loss: 0.5395 - val_recall: 0.4398 Epoch 29/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1638 - recall: 0.7802 - val_loss: 0.5532 - val_recall: 0.4373 Epoch 30/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1651 - recall: 0.7861 - val_loss: 0.5591 - val_recall: 0.4373 Epoch 31/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1568 - recall: 0.7920 - val_loss: 0.5604 - val_recall: 0.4398 Epoch 32/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1547 - recall: 0.8044 - val_loss: 0.5881 - val_recall: 0.4447 Epoch 33/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1583 - recall: 0.7999 - val_loss: 0.5882 - val_recall: 0.4472 Epoch 34/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1459 - recall: 0.8184 - val_loss: 0.6011 - val_recall: 0.4275 Epoch 35/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1393 - recall: 0.8176 - val_loss: 0.6113 - val_recall: 0.4423 Epoch 36/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1364 - recall: 0.8239 - val_loss: 0.6347 - val_recall: 0.4398 Epoch 37/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1406 - recall: 0.8190 - val_loss: 0.6339 - val_recall: 0.4496 Epoch 38/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1350 - recall: 0.8273 - val_loss: 0.6184 - val_recall: 0.4398 Epoch 39/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1352 - recall: 0.8240 - val_loss: 0.6592 - val_recall: 0.4373 Epoch 40/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1401 - recall: 0.8224 - val_loss: 0.6691 - val_recall: 0.4251 Epoch 41/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1246 - recall: 0.8406 - val_loss: 0.6608 - val_recall: 0.4472 Epoch 42/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1223 - recall: 0.8458 - val_loss: 0.6857 - val_recall: 0.4300 Epoch 43/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1228 - recall: 0.8430 - val_loss: 0.6779 - val_recall: 0.4201 Epoch 44/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1132 - recall: 0.8558 - val_loss: 0.7129 - val_recall: 0.4275 Epoch 45/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1071 - recall: 0.8676 - val_loss: 0.7156 - val_recall: 0.4079 Epoch 46/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1071 - recall: 0.8707 - val_loss: 0.7528 - val_recall: 0.4496 Epoch 47/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1025 - recall: 0.8780 - val_loss: 0.7649 - val_recall: 0.4152 Epoch 48/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1092 - recall: 0.8712 - val_loss: 0.7722 - val_recall: 0.4275 Epoch 49/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1142 - recall: 0.8612 - val_loss: 0.7864 - val_recall: 0.4177 Epoch 50/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.1013 - recall: 0.8678 - val_loss: 0.8063 - val_recall: 0.4619
#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_batch_history.history['loss'],adam_batch_history.history['val_loss'])
- The loss in the validation and train are drastically divergent after epoch 4 or so. Small evidence of noise in both sets.
#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(adam_batch_history.history['recall'],adam_batch_history.history['val_recall'])
#Predicting the results using best as a threshold
y_train_pred = adam_batch_model.predict(X_train)
y_train_pred = (y_train_pred > 0.5)
y_train_pred
188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step
array([[False],
[ True],
[ True],
...,
[ True],
[False],
[ True]])
#Predicting the results using 0.5 as the threshold.
y_val_pred = adam_batch_model.predict(X_val)
y_val_pred = (y_val_pred > 0.5)
y_val_pred
63/63 āāāāāāāāāāāāāāāāāāāā 0s 623us/step
array([[False],
[False],
[False],
...,
[False],
[False],
[False]])
#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)
print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])
recall 0.683565 Name: Neural Network (64,32,16,8) with Adam & Batch Normalization, dtype: float64 recall 0.461916 Name: Neural Network (64,32,16,8) with Adam & Batch Normalization, dtype: float64
Classification report
#classification report
print(classification_report(y_train,y_train_pred))
precision recall f1-score support
0 0.92 0.95 0.93 4777
1 0.77 0.68 0.72 1223
accuracy 0.89 6000
macro avg 0.85 0.82 0.83 6000
weighted avg 0.89 0.89 0.89 6000
#classification report
print(classification_report(y_val,y_val_pred))
precision recall f1-score support
0 0.87 0.90 0.88 1593
1 0.54 0.46 0.50 407
accuracy 0.81 2000
macro avg 0.71 0.68 0.69 2000
weighted avg 0.80 0.81 0.81 2000
Confusion matrix
#Calculating the confusion matrix
make_confusion_matrix(y_train, y_train_pred)
#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)
Observations¶
- Batch Normalization didn't do much to help here, still pretty bad recall performance.
- Lets combine with dropout for the next model.
Neural Network (64,32,16,8) with Adam, Batch Normalization & Dropout (0.4, 0.3, 0.2, 0.1)¶
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)
model_name = "Neural Network (64,32,16,8) with Adam, Batch Normalization & Dropout (0.4, 0.3, 0.2, 0.1)"
#Initializing the neural network
adam_batch_dropout_model = Sequential()
adam_batch_dropout_model.add(Dense(64,activation='relu',input_dim = X_train.shape[1]))
adam_batch_dropout_model.add(BatchNormalization())
adam_batch_dropout_model.add(Dropout(0.4))
adam_batch_dropout_model.add(Dense(32,activation='relu'))
adam_batch_dropout_model.add(BatchNormalization())
adam_batch_dropout_model.add(Dropout(0.3))
adam_batch_dropout_model.add(Dense(16,activation='relu'))
adam_batch_dropout_model.add(BatchNormalization())
adam_batch_dropout_model.add(Dropout(0.2))
adam_batch_dropout_model.add(Dense(8,activation='relu'))
adam_batch_dropout_model.add(BatchNormalization())
adam_batch_dropout_model.add(Dropout(0.1))
adam_batch_dropout_model.add(Dense(1, activation = 'sigmoid'))
# use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam()
# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
## compile the model with binary cross entropy as loss function and recall as the metric.
adam_batch_dropout_model.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
# Summary of the model
adam_batch_dropout_model.summary()
Model: "sequential"
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāā ā Layer (type) ā Output Shape ā Param # ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā dense (Dense) ā (None, 64) ā 768 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā batch_normalization ā (None, 64) ā 256 ā ā (BatchNormalization) ā ā ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout (Dropout) ā (None, 64) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_1 (Dense) ā (None, 32) ā 2,080 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā batch_normalization_1 ā (None, 32) ā 128 ā ā (BatchNormalization) ā ā ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_1 (Dropout) ā (None, 32) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_2 (Dense) ā (None, 16) ā 528 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā batch_normalization_2 ā (None, 16) ā 64 ā ā (BatchNormalization) ā ā ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_2 (Dropout) ā (None, 16) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_3 (Dense) ā (None, 8) ā 136 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā batch_normalization_3 ā (None, 8) ā 32 ā ā (BatchNormalization) ā ā ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_3 (Dropout) ā (None, 8) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_4 (Dense) ā (None, 1) ā 9 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāā
Total params: 4,001 (15.63 KB)
Trainable params: 3,761 (14.69 KB)
Non-trainable params: 240 (960.00 B)
Loss function
#Fitting the ANN
adam_batch_dropout_history = adam_batch_dropout_model.fit(
X_train,y_train,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data=(X_val,y_val)
)
Epoch 1/50 94/94 āāāāāāāāāāāāāāāāāāāā 3s 5ms/step - loss: 0.7578 - recall: 0.4486 - val_loss: 0.5735 - val_recall: 0.1106 Epoch 2/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.6124 - recall: 0.4395 - val_loss: 0.5079 - val_recall: 0.1597 Epoch 3/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.5428 - recall: 0.3831 - val_loss: 0.4586 - val_recall: 0.1892 Epoch 4/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.4917 - recall: 0.3463 - val_loss: 0.4274 - val_recall: 0.2678 Epoch 5/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.4627 - recall: 0.3197 - val_loss: 0.4091 - val_recall: 0.2604 Epoch 6/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.4371 - recall: 0.2924 - val_loss: 0.3958 - val_recall: 0.2875 Epoch 7/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.4320 - recall: 0.2972 - val_loss: 0.3866 - val_recall: 0.3194 Epoch 8/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.4202 - recall: 0.3022 - val_loss: 0.3802 - val_recall: 0.2998 Epoch 9/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.4161 - recall: 0.3153 - val_loss: 0.3772 - val_recall: 0.3120 Epoch 10/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.4114 - recall: 0.2951 - val_loss: 0.3701 - val_recall: 0.3784 Epoch 11/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.4091 - recall: 0.3030 - val_loss: 0.3682 - val_recall: 0.3538 Epoch 12/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3996 - recall: 0.3228 - val_loss: 0.3654 - val_recall: 0.3268 Epoch 13/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3951 - recall: 0.3267 - val_loss: 0.3616 - val_recall: 0.3612 Epoch 14/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3903 - recall: 0.3538 - val_loss: 0.3605 - val_recall: 0.3391 Epoch 15/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3870 - recall: 0.3623 - val_loss: 0.3592 - val_recall: 0.3514 Epoch 16/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3868 - recall: 0.3482 - val_loss: 0.3587 - val_recall: 0.3464 Epoch 17/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3875 - recall: 0.3509 - val_loss: 0.3581 - val_recall: 0.3710 Epoch 18/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3867 - recall: 0.3550 - val_loss: 0.3595 - val_recall: 0.3612 Epoch 19/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3828 - recall: 0.3429 - val_loss: 0.3583 - val_recall: 0.3661 Epoch 20/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3796 - recall: 0.3467 - val_loss: 0.3585 - val_recall: 0.3415 Epoch 21/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3765 - recall: 0.3724 - val_loss: 0.3580 - val_recall: 0.3415 Epoch 22/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3686 - recall: 0.3711 - val_loss: 0.3570 - val_recall: 0.3538 Epoch 23/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3770 - recall: 0.3799 - val_loss: 0.3559 - val_recall: 0.3538 Epoch 24/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3740 - recall: 0.3685 - val_loss: 0.3568 - val_recall: 0.3440 Epoch 25/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3677 - recall: 0.3700 - val_loss: 0.3569 - val_recall: 0.3514 Epoch 26/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3636 - recall: 0.3824 - val_loss: 0.3565 - val_recall: 0.3563 Epoch 27/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3646 - recall: 0.4192 - val_loss: 0.3550 - val_recall: 0.3636 Epoch 28/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3580 - recall: 0.4192 - val_loss: 0.3555 - val_recall: 0.3636 Epoch 29/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3649 - recall: 0.3852 - val_loss: 0.3554 - val_recall: 0.3661 Epoch 30/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3653 - recall: 0.4026 - val_loss: 0.3557 - val_recall: 0.3636 Epoch 31/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3574 - recall: 0.4182 - val_loss: 0.3580 - val_recall: 0.3710 Epoch 32/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3565 - recall: 0.4211 - val_loss: 0.3582 - val_recall: 0.3784 Epoch 33/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3681 - recall: 0.3878 - val_loss: 0.3556 - val_recall: 0.3833 Epoch 34/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3581 - recall: 0.4149 - val_loss: 0.3551 - val_recall: 0.3759 Epoch 35/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3592 - recall: 0.3985 - val_loss: 0.3562 - val_recall: 0.3661 Epoch 36/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3643 - recall: 0.3872 - val_loss: 0.3560 - val_recall: 0.3563 Epoch 37/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3626 - recall: 0.4112 - val_loss: 0.3551 - val_recall: 0.3489 Epoch 38/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3572 - recall: 0.4158 - val_loss: 0.3555 - val_recall: 0.3636 Epoch 39/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3573 - recall: 0.4160 - val_loss: 0.3554 - val_recall: 0.3735 Epoch 40/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3530 - recall: 0.4301 - val_loss: 0.3552 - val_recall: 0.3808 Epoch 41/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3611 - recall: 0.3974 - val_loss: 0.3567 - val_recall: 0.3735 Epoch 42/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3608 - recall: 0.4059 - val_loss: 0.3588 - val_recall: 0.3612 Epoch 43/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3578 - recall: 0.4137 - val_loss: 0.3575 - val_recall: 0.3587 Epoch 44/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3538 - recall: 0.4136 - val_loss: 0.3549 - val_recall: 0.3956 Epoch 45/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3584 - recall: 0.4348 - val_loss: 0.3558 - val_recall: 0.3735 Epoch 46/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3490 - recall: 0.4300 - val_loss: 0.3580 - val_recall: 0.3686 Epoch 47/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3481 - recall: 0.4219 - val_loss: 0.3572 - val_recall: 0.3808 Epoch 48/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3531 - recall: 0.4404 - val_loss: 0.3566 - val_recall: 0.3759 Epoch 49/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3543 - recall: 0.4281 - val_loss: 0.3575 - val_recall: 0.3931 Epoch 50/50 94/94 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.3560 - recall: 0.4223 - val_loss: 0.3580 - val_recall: 0.3661
#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_batch_dropout_history.history['loss'],adam_batch_dropout_history.history['val_loss'])
- The loss in the validation and train are similar, and begin converging around epoch 38, and some noise in both sets though much more in training due to dropout
#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(adam_batch_dropout_history.history['recall'],adam_batch_dropout_history.history['val_recall'])
#Predicting the results using best as a threshold
y_train_pred = adam_batch_dropout_model.predict(X_train)
y_train_pred = (y_train_pred > 0.5)
y_train_pred
188/188 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step
array([[False],
[False],
[False],
...,
[ True],
[False],
[ True]])
#Predicting the results using 0.5 as the threshold.
y_val_pred = adam_batch_dropout_model.predict(X_val)
y_val_pred = (y_val_pred > 0.5)
y_val_pred
63/63 āāāāāāāāāāāāāāāāāāāā 0s 617us/step
array([[False],
[False],
[False],
...,
[False],
[False],
[False]])
#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)
print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])
recall 0.413737 Name: Neural Network (64,32,16,8) with Adam, Batch Normalization & Dropout (0.4, 0.3, 0.2, 0.1), dtype: float64 recall 0.366093 Name: Neural Network (64,32,16,8) with Adam, Batch Normalization & Dropout (0.4, 0.3, 0.2, 0.1), dtype: float64
Classification report
#classification report
print(classification_report(y_train,y_train_pred))
precision recall f1-score support
0 0.87 0.98 0.92 4777
1 0.84 0.41 0.55 1223
accuracy 0.86 6000
macro avg 0.85 0.70 0.74 6000
weighted avg 0.86 0.86 0.85 6000
#classification report
print(classification_report(y_val,y_val_pred))
precision recall f1-score support
0 0.86 0.98 0.91 1593
1 0.81 0.37 0.50 407
accuracy 0.85 2000
macro avg 0.83 0.67 0.71 2000
weighted avg 0.85 0.85 0.83 2000
Confusion matrix
#Calculating the confusion matrix
make_confusion_matrix(y_train, y_train_pred)
#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)
Observations¶
- Even worse performance. In this case I do not think batch normalization should be used at all.
- Next model we will retry SGD, but use SMOTE to have better data to train on.
Neural Network (64,32,16,8) with SMOTE & SGD Optimizer¶
sm = SMOTE(random_state=42)
# fit SMOTE on the training data.
X_train_smote, y_train_smote= sm.fit_resample(X_train,y_train)
print('After UpSampling, the shape of train_X: {}'.format(X_train_smote.shape))
print('After UpSampling, the shape of train_y: {} \n'.format(y_train_smote.shape))
After UpSampling, the shape of train_X: (9554, 11) After UpSampling, the shape of train_y: (9554,)
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)
model_name = "Neural Network (64,32,16,8) with SMOTE & SGD"
#Initializing the model
sgd_smote_model = Sequential()
sgd_smote_model.add(Dense(64,activation='relu',input_dim = X_train_smote.shape[1]))
sgd_smote_model.add(Dense(32,activation='relu'))
sgd_smote_model.add(Dense(16,activation='relu'))
sgd_smote_model.add(Dense(8,activation='relu'))
sgd_smote_model.add(Dense(1, activation = 'sigmoid'))
# use SGD as the optimizer.
optimizer = tf.keras.optimizers.SGD()
# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
# compile the model with binary cross entropy as loss function and recall as the metric
sgd_smote_model.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
sgd_smote_model.summary()
Model: "sequential"
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāā ā Layer (type) ā Output Shape ā Param # ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā dense (Dense) ā (None, 64) ā 768 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_1 (Dense) ā (None, 32) ā 2,080 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_2 (Dense) ā (None, 16) ā 528 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_3 (Dense) ā (None, 8) ā 136 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_4 (Dense) ā (None, 1) ā 9 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāā
Total params: 3,521 (13.75 KB)
Trainable params: 3,521 (13.75 KB)
Non-trainable params: 0 (0.00 B)
#Fitting the ANN
sgd_smote_history = sgd_smote_model.fit(
X_train_smote, y_train_smote,
batch_size=batch_size,
epochs=epochs,
#class_weight=cw_dict,
verbose=1,
validation_data = (X_val,y_val)
)
Epoch 1/50 150/150 āāāāāāāāāāāāāāāāāāāā 1s 2ms/step - loss: 0.6940 - recall: 0.3086 - val_loss: 0.6787 - val_recall: 0.5332 Epoch 2/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 984us/step - loss: 0.6782 - recall: 0.5645 - val_loss: 0.6559 - val_recall: 0.6118 Epoch 3/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 998us/step - loss: 0.6551 - recall: 0.5830 - val_loss: 0.6205 - val_recall: 0.6658 Epoch 4/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 991us/step - loss: 0.6234 - recall: 0.6380 - val_loss: 0.5968 - val_recall: 0.7174 Epoch 5/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 981us/step - loss: 0.5951 - recall: 0.6714 - val_loss: 0.5836 - val_recall: 0.7273 Epoch 6/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5763 - recall: 0.6981 - val_loss: 0.5781 - val_recall: 0.7322 Epoch 7/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 986us/step - loss: 0.5652 - recall: 0.7168 - val_loss: 0.5760 - val_recall: 0.7396 Epoch 8/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5571 - recall: 0.7283 - val_loss: 0.5725 - val_recall: 0.7445 Epoch 9/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5495 - recall: 0.7338 - val_loss: 0.5671 - val_recall: 0.7469 Epoch 10/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5418 - recall: 0.7401 - val_loss: 0.5644 - val_recall: 0.7469 Epoch 11/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 990us/step - loss: 0.5345 - recall: 0.7445 - val_loss: 0.5614 - val_recall: 0.7543 Epoch 12/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 970us/step - loss: 0.5278 - recall: 0.7493 - val_loss: 0.5578 - val_recall: 0.7543 Epoch 13/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 979us/step - loss: 0.5213 - recall: 0.7544 - val_loss: 0.5546 - val_recall: 0.7518 Epoch 14/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5149 - recall: 0.7547 - val_loss: 0.5505 - val_recall: 0.7617 Epoch 15/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 980us/step - loss: 0.5085 - recall: 0.7595 - val_loss: 0.5475 - val_recall: 0.7641 Epoch 16/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 984us/step - loss: 0.5023 - recall: 0.7630 - val_loss: 0.5415 - val_recall: 0.7690 Epoch 17/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 962us/step - loss: 0.4959 - recall: 0.7669 - val_loss: 0.5362 - val_recall: 0.7690 Epoch 18/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 968us/step - loss: 0.4898 - recall: 0.7704 - val_loss: 0.5300 - val_recall: 0.7715 Epoch 19/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 981us/step - loss: 0.4837 - recall: 0.7720 - val_loss: 0.5239 - val_recall: 0.7764 Epoch 20/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 986us/step - loss: 0.4780 - recall: 0.7706 - val_loss: 0.5193 - val_recall: 0.7813 Epoch 21/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 994us/step - loss: 0.4728 - recall: 0.7695 - val_loss: 0.5132 - val_recall: 0.7764 Epoch 22/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 988us/step - loss: 0.4680 - recall: 0.7711 - val_loss: 0.5075 - val_recall: 0.7764 Epoch 23/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4638 - recall: 0.7704 - val_loss: 0.5031 - val_recall: 0.7715 Epoch 24/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4600 - recall: 0.7698 - val_loss: 0.4988 - val_recall: 0.7715 Epoch 25/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 963us/step - loss: 0.4566 - recall: 0.7729 - val_loss: 0.4953 - val_recall: 0.7715 Epoch 26/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 969us/step - loss: 0.4534 - recall: 0.7737 - val_loss: 0.4919 - val_recall: 0.7740 Epoch 27/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 974us/step - loss: 0.4506 - recall: 0.7738 - val_loss: 0.4888 - val_recall: 0.7715 Epoch 28/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 962us/step - loss: 0.4480 - recall: 0.7743 - val_loss: 0.4856 - val_recall: 0.7641 Epoch 29/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 981us/step - loss: 0.4456 - recall: 0.7751 - val_loss: 0.4825 - val_recall: 0.7592 Epoch 30/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4432 - recall: 0.7744 - val_loss: 0.4805 - val_recall: 0.7592 Epoch 31/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4410 - recall: 0.7746 - val_loss: 0.4785 - val_recall: 0.7592 Epoch 32/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 990us/step - loss: 0.4388 - recall: 0.7755 - val_loss: 0.4756 - val_recall: 0.7518 Epoch 33/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 988us/step - loss: 0.4367 - recall: 0.7776 - val_loss: 0.4732 - val_recall: 0.7494 Epoch 34/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4347 - recall: 0.7780 - val_loss: 0.4716 - val_recall: 0.7494 Epoch 35/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4327 - recall: 0.7794 - val_loss: 0.4697 - val_recall: 0.7494 Epoch 36/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4307 - recall: 0.7800 - val_loss: 0.4683 - val_recall: 0.7494 Epoch 37/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4289 - recall: 0.7819 - val_loss: 0.4665 - val_recall: 0.7469 Epoch 38/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 982us/step - loss: 0.4271 - recall: 0.7830 - val_loss: 0.4650 - val_recall: 0.7445 Epoch 39/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 970us/step - loss: 0.4255 - recall: 0.7840 - val_loss: 0.4642 - val_recall: 0.7469 Epoch 40/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4240 - recall: 0.7839 - val_loss: 0.4632 - val_recall: 0.7445 Epoch 41/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4225 - recall: 0.7858 - val_loss: 0.4626 - val_recall: 0.7420 Epoch 42/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 995us/step - loss: 0.4211 - recall: 0.7858 - val_loss: 0.4620 - val_recall: 0.7396 Epoch 43/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 983us/step - loss: 0.4197 - recall: 0.7861 - val_loss: 0.4606 - val_recall: 0.7396 Epoch 44/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 990us/step - loss: 0.4183 - recall: 0.7877 - val_loss: 0.4592 - val_recall: 0.7396 Epoch 45/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 980us/step - loss: 0.4170 - recall: 0.7872 - val_loss: 0.4582 - val_recall: 0.7322 Epoch 46/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 980us/step - loss: 0.4158 - recall: 0.7891 - val_loss: 0.4569 - val_recall: 0.7248 Epoch 47/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4145 - recall: 0.7878 - val_loss: 0.4560 - val_recall: 0.7248 Epoch 48/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4133 - recall: 0.7875 - val_loss: 0.4555 - val_recall: 0.7248 Epoch 49/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 982us/step - loss: 0.4121 - recall: 0.7913 - val_loss: 0.4550 - val_recall: 0.7248 Epoch 50/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 979us/step - loss: 0.4110 - recall: 0.7929 - val_loss: 0.4550 - val_recall: 0.7273
Loss function
#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(sgd_smote_history.history['loss'],sgd_smote_history.history['val_loss'])
- The loss in the validation and train are similar, though divergent, with less noise evident.
#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(sgd_smote_history.history['recall'],sgd_smote_history.history['val_recall'])
y_train_pred = sgd_smote_model.predict(X_train_smote)
#Predicting the results using 0.5 as the threshold
y_train_pred = (y_train_pred > 0.5)
y_train_pred
299/299 āāāāāāāāāāāāāāāāāāāā 0s 647us/step
array([[ True],
[ True],
[ True],
...,
[ True],
[False],
[ True]])
y_val_pred = sgd_smote_model.predict(X_val)
#Predicting the results using 0.5 as the threshold
y_val_pred = (y_val_pred > 0.5)
y_val_pred
63/63 āāāāāāāāāāāāāāāāāāāā 0s 559us/step
array([[False],
[False],
[ True],
...,
[False],
[False],
[ True]])
#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train_smote,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)
print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])
recall 0.805526 Name: Neural Network (64,32,16,8) with SMOTE & SGD, dtype: float64 recall 0.727273 Name: Neural Network (64,32,16,8) with SMOTE & SGD, dtype: float64
Classification report
print(classification_report(y_train_smote,y_train_pred))
precision recall f1-score support
0 0.81 0.82 0.81 4777
1 0.82 0.81 0.81 4777
accuracy 0.81 9554
macro avg 0.81 0.81 0.81 9554
weighted avg 0.81 0.81 0.81 9554
print(classification_report(y_val,y_val_pred))
precision recall f1-score support
0 0.92 0.80 0.86 1593
1 0.49 0.73 0.58 407
accuracy 0.79 2000
macro avg 0.70 0.77 0.72 2000
weighted avg 0.83 0.79 0.80 2000
Confusion matrix
#Calculating the confusion matrix
make_confusion_matrix(y_train_smote, y_train_pred)
#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)
Observations¶
- Recall scores improving.
- Lets try with Adam.
Neural Network (64,32,16,8) with SMOTE & Adam Optimizer¶
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)
model_name = "Neural Network (64,32,16,8) with SMOTE & Adam"
#Initializing the model
adam_smote_model = Sequential()
adam_smote_model.add(Dense(64,activation='relu',input_dim = X_train_smote.shape[1]))
adam_smote_model.add(Dense(32,activation='relu'))
adam_smote_model.add(Dense(16,activation='relu'))
adam_smote_model.add(Dense(8,activation='relu'))
adam_smote_model.add(Dense(1, activation = 'sigmoid'))
adam_smote_model.summary()
Model: "sequential"
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāā ā Layer (type) ā Output Shape ā Param # ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā dense (Dense) ā (None, 64) ā 768 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_1 (Dense) ā (None, 32) ā 2,080 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_2 (Dense) ā (None, 16) ā 528 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_3 (Dense) ā (None, 8) ā 136 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_4 (Dense) ā (None, 1) ā 9 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāā
Total params: 3,521 (13.75 KB)
Trainable params: 3,521 (13.75 KB)
Non-trainable params: 0 (0.00 B)
# use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam()
# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
# compile the model with binary cross entropy as loss function and recall as the metric
adam_smote_model.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
adam_smote_model.summary()
Model: "sequential"
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāā ā Layer (type) ā Output Shape ā Param # ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā dense (Dense) ā (None, 64) ā 768 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_1 (Dense) ā (None, 32) ā 2,080 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_2 (Dense) ā (None, 16) ā 528 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_3 (Dense) ā (None, 8) ā 136 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_4 (Dense) ā (None, 1) ā 9 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāā
Total params: 3,521 (13.75 KB)
Trainable params: 3,521 (13.75 KB)
Non-trainable params: 0 (0.00 B)
#Fitting the ANN
adam_smote_history = adam_smote_model.fit(
X_train_smote,y_train_smote,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data = (X_val,y_val)
)
Epoch 1/50 150/150 āāāāāāāāāāāāāāāāāāāā 1s 2ms/step - loss: 0.6328 - recall: 0.5413 - val_loss: 0.5316 - val_recall: 0.7346 Epoch 2/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5091 - recall: 0.7609 - val_loss: 0.4842 - val_recall: 0.7543 Epoch 3/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4638 - recall: 0.7730 - val_loss: 0.4608 - val_recall: 0.7346 Epoch 4/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4445 - recall: 0.7807 - val_loss: 0.4580 - val_recall: 0.7224 Epoch 5/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4318 - recall: 0.7911 - val_loss: 0.4555 - val_recall: 0.7076 Epoch 6/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4211 - recall: 0.7970 - val_loss: 0.4546 - val_recall: 0.7150 Epoch 7/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4126 - recall: 0.8052 - val_loss: 0.4568 - val_recall: 0.7125 Epoch 8/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4043 - recall: 0.8111 - val_loss: 0.4567 - val_recall: 0.7125 Epoch 9/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3971 - recall: 0.8144 - val_loss: 0.4569 - val_recall: 0.7002 Epoch 10/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3902 - recall: 0.8181 - val_loss: 0.4597 - val_recall: 0.7027 Epoch 11/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3835 - recall: 0.8271 - val_loss: 0.4669 - val_recall: 0.7101 Epoch 12/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3769 - recall: 0.8310 - val_loss: 0.4701 - val_recall: 0.7076 Epoch 13/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3708 - recall: 0.8386 - val_loss: 0.4712 - val_recall: 0.7125 Epoch 14/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3647 - recall: 0.8414 - val_loss: 0.4743 - val_recall: 0.7052 Epoch 15/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3583 - recall: 0.8468 - val_loss: 0.4742 - val_recall: 0.6978 Epoch 16/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3526 - recall: 0.8488 - val_loss: 0.4796 - val_recall: 0.6953 Epoch 17/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3463 - recall: 0.8552 - val_loss: 0.4841 - val_recall: 0.6929 Epoch 18/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3409 - recall: 0.8573 - val_loss: 0.4881 - val_recall: 0.6880 Epoch 19/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3367 - recall: 0.8610 - val_loss: 0.4962 - val_recall: 0.6929 Epoch 20/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3314 - recall: 0.8665 - val_loss: 0.4940 - val_recall: 0.6929 Epoch 21/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3270 - recall: 0.8689 - val_loss: 0.4986 - val_recall: 0.6880 Epoch 22/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3230 - recall: 0.8742 - val_loss: 0.5001 - val_recall: 0.6904 Epoch 23/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3177 - recall: 0.8742 - val_loss: 0.5062 - val_recall: 0.6929 Epoch 24/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3136 - recall: 0.8789 - val_loss: 0.5107 - val_recall: 0.6929 Epoch 25/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3094 - recall: 0.8854 - val_loss: 0.5137 - val_recall: 0.6904 Epoch 26/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3049 - recall: 0.8883 - val_loss: 0.5238 - val_recall: 0.6953 Epoch 27/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.3003 - recall: 0.8889 - val_loss: 0.5297 - val_recall: 0.6855 Epoch 28/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2966 - recall: 0.8918 - val_loss: 0.5304 - val_recall: 0.6806 Epoch 29/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2926 - recall: 0.8939 - val_loss: 0.5325 - val_recall: 0.6732 Epoch 30/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2888 - recall: 0.8954 - val_loss: 0.5390 - val_recall: 0.6732 Epoch 31/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2855 - recall: 0.8988 - val_loss: 0.5455 - val_recall: 0.6683 Epoch 32/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2820 - recall: 0.9005 - val_loss: 0.5497 - val_recall: 0.6708 Epoch 33/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2797 - recall: 0.9014 - val_loss: 0.5564 - val_recall: 0.6683 Epoch 34/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2758 - recall: 0.9054 - val_loss: 0.5634 - val_recall: 0.6732 Epoch 35/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2736 - recall: 0.9070 - val_loss: 0.5692 - val_recall: 0.6732 Epoch 36/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2710 - recall: 0.9087 - val_loss: 0.5763 - val_recall: 0.6757 Epoch 37/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2685 - recall: 0.9096 - val_loss: 0.5781 - val_recall: 0.6708 Epoch 38/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2655 - recall: 0.9126 - val_loss: 0.5816 - val_recall: 0.6732 Epoch 39/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2622 - recall: 0.9152 - val_loss: 0.5890 - val_recall: 0.6732 Epoch 40/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2588 - recall: 0.9166 - val_loss: 0.5959 - val_recall: 0.6683 Epoch 41/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2565 - recall: 0.9180 - val_loss: 0.6005 - val_recall: 0.6560 Epoch 42/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2545 - recall: 0.9176 - val_loss: 0.6070 - val_recall: 0.6634 Epoch 43/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2520 - recall: 0.9182 - val_loss: 0.6118 - val_recall: 0.6536 Epoch 44/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2492 - recall: 0.9162 - val_loss: 0.6176 - val_recall: 0.6560 Epoch 45/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2471 - recall: 0.9169 - val_loss: 0.6258 - val_recall: 0.6634 Epoch 46/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2441 - recall: 0.9177 - val_loss: 0.6342 - val_recall: 0.6634 Epoch 47/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2422 - recall: 0.9195 - val_loss: 0.6396 - val_recall: 0.6585 Epoch 48/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2410 - recall: 0.9201 - val_loss: 0.6448 - val_recall: 0.6560 Epoch 49/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2389 - recall: 0.9234 - val_loss: 0.6539 - val_recall: 0.6560 Epoch 50/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.2362 - recall: 0.9260 - val_loss: 0.6539 - val_recall: 0.6511
Loss function
#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_smote_history.history['loss'],adam_smote_history.history['val_loss'])
- The loss in the validation and train diverge after the first couple of epochs. Small amount of noise evident in validation, while training is smooth.
#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(adam_smote_history.history['recall'],adam_smote_history.history['val_recall'])
y_train_pred = adam_smote_model.predict(X_train_smote)
#Predicting the results using 0.5 as the threshold
y_train_pred = (y_train_pred > 0.5)
y_train_pred
299/299 āāāāāāāāāāāāāāāāāāāā 0s 660us/step
array([[False],
[ True],
[ True],
...,
[ True],
[False],
[ True]])
y_val_pred = adam_smote_model.predict(X_val)
#Predicting the results using 0.5 as the threshold
y_val_pred = (y_val_pred > 0.5)
y_val_pred
63/63 āāāāāāāāāāāāāāāāāāāā 0s 550us/step
array([[False],
[False],
[ True],
...,
[False],
[False],
[False]])
#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train_smote,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)
print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])
recall 0.941386 Name: Neural Network (64,32,16,8) with SMOTE & Adam, dtype: float64 recall 0.651106 Name: Neural Network (64,32,16,8) with SMOTE & Adam, dtype: float64
Classification report
print(classification_report(y_train_smote,y_train_pred))
precision recall f1-score support
0 0.94 0.88 0.91 4777
1 0.88 0.94 0.91 4777
accuracy 0.91 9554
macro avg 0.91 0.91 0.91 9554
weighted avg 0.91 0.91 0.91 9554
print(classification_report(y_val,y_val_pred))
precision recall f1-score support
0 0.90 0.82 0.86 1593
1 0.48 0.65 0.56 407
accuracy 0.79 2000
macro avg 0.69 0.74 0.71 2000
weighted avg 0.82 0.79 0.80 2000
Confusion matrix
#Calculating the confusion matrix
make_confusion_matrix(y_train_smote, y_train_pred)
#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)
Observations¶
- Good performance on training set recall, but vlaidation is going the wrong way.
- Lets see if adding dropout back in helps.
Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, and Dropout¶
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)
model_name = "Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1)"
#Initializing the model
adam_dropout_smote_model001 = Sequential()
adam_dropout_smote_model001.add(Dense(64,activation='relu',input_dim = X_train_smote.shape[1]))
adam_dropout_smote_model001.add(Dropout(0.8))
adam_dropout_smote_model001.add(Dense(32,activation='relu'))
adam_dropout_smote_model001.add(Dropout(0.4))
adam_dropout_smote_model001.add(Dense(16,activation='relu'))
adam_dropout_smote_model001.add(Dropout(0.2))
adam_dropout_smote_model001.add(Dense(8,activation='relu'))
adam_dropout_smote_model001.add(Dropout(0.1))
adam_dropout_smote_model001.add(Dense(1, activation = 'sigmoid'))
# use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam()
# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
# compile the model with binary cross entropy as loss function and recall as the metric
adam_dropout_smote_model001.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
adam_dropout_smote_model001.summary()
Model: "sequential"
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāā ā Layer (type) ā Output Shape ā Param # ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā dense (Dense) ā (None, 64) ā 768 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout (Dropout) ā (None, 64) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_1 (Dense) ā (None, 32) ā 2,080 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_1 (Dropout) ā (None, 32) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_2 (Dense) ā (None, 16) ā 528 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_2 (Dropout) ā (None, 16) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_3 (Dense) ā (None, 8) ā 136 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_3 (Dropout) ā (None, 8) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_4 (Dense) ā (None, 1) ā 9 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāā
Total params: 3,521 (13.75 KB)
Trainable params: 3,521 (13.75 KB)
Non-trainable params: 0 (0.00 B)
adam_dropout_smote_history001 = adam_dropout_smote_model001.fit(
X_train_smote,y_train_smote,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data = (X_val,y_val))
Epoch 1/50 150/150 āāāāāāāāāāāāāāāāāāāā 2s 3ms/step - loss: 0.6936 - recall: 0.5652 - val_loss: 0.6665 - val_recall: 0.4373 Epoch 2/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.6737 - recall: 0.5789 - val_loss: 0.6285 - val_recall: 0.6658 Epoch 3/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.6447 - recall: 0.6325 - val_loss: 0.5874 - val_recall: 0.6757 Epoch 4/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.6238 - recall: 0.6817 - val_loss: 0.5840 - val_recall: 0.6929 Epoch 5/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.6102 - recall: 0.7034 - val_loss: 0.5659 - val_recall: 0.6806 Epoch 6/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5994 - recall: 0.7122 - val_loss: 0.5747 - val_recall: 0.6929 Epoch 7/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5941 - recall: 0.7199 - val_loss: 0.5692 - val_recall: 0.7199 Epoch 8/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5940 - recall: 0.7292 - val_loss: 0.5688 - val_recall: 0.7322 Epoch 9/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5749 - recall: 0.7200 - val_loss: 0.5651 - val_recall: 0.7322 Epoch 10/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5747 - recall: 0.7319 - val_loss: 0.5509 - val_recall: 0.7248 Epoch 11/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5621 - recall: 0.7398 - val_loss: 0.5479 - val_recall: 0.7273 Epoch 12/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5668 - recall: 0.7204 - val_loss: 0.5484 - val_recall: 0.7224 Epoch 13/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5552 - recall: 0.7190 - val_loss: 0.5431 - val_recall: 0.7445 Epoch 14/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5545 - recall: 0.7339 - val_loss: 0.5368 - val_recall: 0.7396 Epoch 15/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5509 - recall: 0.7205 - val_loss: 0.5186 - val_recall: 0.7420 Epoch 16/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5478 - recall: 0.7267 - val_loss: 0.5284 - val_recall: 0.7641 Epoch 17/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5384 - recall: 0.7392 - val_loss: 0.5158 - val_recall: 0.7690 Epoch 18/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5382 - recall: 0.7329 - val_loss: 0.5072 - val_recall: 0.7543 Epoch 19/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5350 - recall: 0.7327 - val_loss: 0.5163 - val_recall: 0.7764 Epoch 20/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5301 - recall: 0.7459 - val_loss: 0.5057 - val_recall: 0.7518 Epoch 21/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5264 - recall: 0.7420 - val_loss: 0.5016 - val_recall: 0.7789 Epoch 22/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5235 - recall: 0.7550 - val_loss: 0.5136 - val_recall: 0.7887 Epoch 23/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5246 - recall: 0.7442 - val_loss: 0.4978 - val_recall: 0.7690 Epoch 24/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5181 - recall: 0.7430 - val_loss: 0.4960 - val_recall: 0.7666 Epoch 25/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5240 - recall: 0.7515 - val_loss: 0.5105 - val_recall: 0.8059 Epoch 26/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5139 - recall: 0.7613 - val_loss: 0.5018 - val_recall: 0.8108 Epoch 27/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5099 - recall: 0.7471 - val_loss: 0.5035 - val_recall: 0.7936 Epoch 28/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5132 - recall: 0.7586 - val_loss: 0.5054 - val_recall: 0.8059 Epoch 29/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5138 - recall: 0.7538 - val_loss: 0.5024 - val_recall: 0.7912 Epoch 30/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5065 - recall: 0.7623 - val_loss: 0.4942 - val_recall: 0.7764 Epoch 31/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5091 - recall: 0.7516 - val_loss: 0.4914 - val_recall: 0.7592 Epoch 32/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5091 - recall: 0.7583 - val_loss: 0.4863 - val_recall: 0.7789 Epoch 33/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5043 - recall: 0.7657 - val_loss: 0.4869 - val_recall: 0.7617 Epoch 34/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4994 - recall: 0.7532 - val_loss: 0.4919 - val_recall: 0.7592 Epoch 35/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5031 - recall: 0.7582 - val_loss: 0.4884 - val_recall: 0.7813 Epoch 36/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5006 - recall: 0.7559 - val_loss: 0.4807 - val_recall: 0.7617 Epoch 37/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4987 - recall: 0.7628 - val_loss: 0.4800 - val_recall: 0.7985 Epoch 38/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5025 - recall: 0.7629 - val_loss: 0.4768 - val_recall: 0.7862 Epoch 39/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4932 - recall: 0.7589 - val_loss: 0.4843 - val_recall: 0.7592 Epoch 40/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5006 - recall: 0.7640 - val_loss: 0.4771 - val_recall: 0.7445 Epoch 41/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4965 - recall: 0.7684 - val_loss: 0.4798 - val_recall: 0.8059 Epoch 42/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4973 - recall: 0.7644 - val_loss: 0.4882 - val_recall: 0.8034 Epoch 43/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4867 - recall: 0.7713 - val_loss: 0.4802 - val_recall: 0.7543 Epoch 44/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5004 - recall: 0.7565 - val_loss: 0.4834 - val_recall: 0.7469 Epoch 45/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4857 - recall: 0.7607 - val_loss: 0.4721 - val_recall: 0.7445 Epoch 46/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4868 - recall: 0.7642 - val_loss: 0.4810 - val_recall: 0.7494 Epoch 47/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4962 - recall: 0.7591 - val_loss: 0.4828 - val_recall: 0.7445 Epoch 48/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4950 - recall: 0.7679 - val_loss: 0.4767 - val_recall: 0.7322 Epoch 49/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4900 - recall: 0.7549 - val_loss: 0.4817 - val_recall: 0.7346 Epoch 50/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4887 - recall: 0.7631 - val_loss: 0.4792 - val_recall: 0.7297
Loss function
#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_dropout_smote_history001.history['loss'],adam_dropout_smote_history001.history['val_loss'])
- The loss in the validation and train are similar, though divergent, and shows a bit of noise in both sets, likely due to dropout and oversampling.
#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(adam_dropout_smote_history001.history['recall'],adam_dropout_smote_history001.history['val_recall'])
y_train_pred = adam_dropout_smote_model001.predict(X_train_smote)
#Predicting the results using 0.5 as the threshold
y_train_pred = (y_train_pred > 0.5)
y_train_pred
299/299 āāāāāāāāāāāāāāāāāāāā 0s 659us/step
array([[ True],
[ True],
[ True],
...,
[ True],
[False],
[ True]])
y_val_pred = adam_dropout_smote_model001.predict(X_val)
#Predicting the results using 0.5 as the threshold
y_val_pred = (y_val_pred > 0.5)
y_val_pred
63/63 āāāāāāāāāāāāāāāāāāāā 0s 565us/step
array([[False],
[False],
[ True],
...,
[False],
[False],
[ True]])
#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train_smote,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)
print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])
recall 0.793804 Name: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1), dtype: float64 recall 0.72973 Name: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1), dtype: float64
Classification report
print(classification_report(y_train_smote,y_train_pred))
precision recall f1-score support
0 0.80 0.81 0.80 4777
1 0.81 0.79 0.80 4777
accuracy 0.80 9554
macro avg 0.80 0.80 0.80 9554
weighted avg 0.80 0.80 0.80 9554
print(classification_report(y_val,y_val_pred))
precision recall f1-score support
0 0.92 0.80 0.86 1593
1 0.48 0.73 0.58 407
accuracy 0.78 2000
macro avg 0.70 0.76 0.72 2000
weighted avg 0.83 0.78 0.80 2000
Confusion matrix
#Calculating the confusion matrix
make_confusion_matrix(y_train_smote, y_train_pred)
#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)
Observations¶
- Dropout seems to have helped improve recall scores.
- Lets try adjusting the Learning rate. We'll start with 0.001
Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.001)¶
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)
model_name = "Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.001)"
#Initializing the model
adam_dropout_smote_model001 = Sequential()
adam_dropout_smote_model001.add(Dense(64,activation='relu',input_dim = X_train_smote.shape[1]))
adam_dropout_smote_model001.add(Dropout(0.8))
adam_dropout_smote_model001.add(Dense(32,activation='relu'))
adam_dropout_smote_model001.add(Dropout(0.4))
adam_dropout_smote_model001.add(Dense(16,activation='relu'))
adam_dropout_smote_model001.add(Dropout(0.2))
adam_dropout_smote_model001.add(Dense(8,activation='relu'))
adam_dropout_smote_model001.add(Dropout(0.1))
adam_dropout_smote_model001.add(Dense(1, activation = 'sigmoid'))
# use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)
# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
# compile the model with binary cross entropy as loss function and recall as the metric
adam_dropout_smote_model001.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
adam_dropout_smote_model001.summary()
Model: "sequential"
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāā ā Layer (type) ā Output Shape ā Param # ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā dense (Dense) ā (None, 64) ā 768 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout (Dropout) ā (None, 64) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_1 (Dense) ā (None, 32) ā 2,080 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_1 (Dropout) ā (None, 32) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_2 (Dense) ā (None, 16) ā 528 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_2 (Dropout) ā (None, 16) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_3 (Dense) ā (None, 8) ā 136 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_3 (Dropout) ā (None, 8) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_4 (Dense) ā (None, 1) ā 9 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāā
Total params: 3,521 (13.75 KB)
Trainable params: 3,521 (13.75 KB)
Non-trainable params: 0 (0.00 B)
adam_dropout_smote_history001 = adam_dropout_smote_model001.fit(
X_train_smote,y_train_smote,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data = (X_val,y_val))
Epoch 1/50 150/150 āāāāāāāāāāāāāāāāāāāā 2s 3ms/step - loss: 0.6936 - recall: 0.5652 - val_loss: 0.6665 - val_recall: 0.4373 Epoch 2/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.6737 - recall: 0.5789 - val_loss: 0.6285 - val_recall: 0.6658 Epoch 3/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.6447 - recall: 0.6325 - val_loss: 0.5874 - val_recall: 0.6757 Epoch 4/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.6238 - recall: 0.6817 - val_loss: 0.5840 - val_recall: 0.6929 Epoch 5/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.6102 - recall: 0.7034 - val_loss: 0.5659 - val_recall: 0.6806 Epoch 6/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5994 - recall: 0.7122 - val_loss: 0.5747 - val_recall: 0.6929 Epoch 7/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5941 - recall: 0.7199 - val_loss: 0.5692 - val_recall: 0.7199 Epoch 8/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5940 - recall: 0.7292 - val_loss: 0.5688 - val_recall: 0.7322 Epoch 9/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5749 - recall: 0.7200 - val_loss: 0.5651 - val_recall: 0.7322 Epoch 10/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5747 - recall: 0.7319 - val_loss: 0.5509 - val_recall: 0.7248 Epoch 11/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5621 - recall: 0.7398 - val_loss: 0.5479 - val_recall: 0.7273 Epoch 12/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5668 - recall: 0.7204 - val_loss: 0.5484 - val_recall: 0.7224 Epoch 13/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5552 - recall: 0.7190 - val_loss: 0.5431 - val_recall: 0.7445 Epoch 14/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5545 - recall: 0.7339 - val_loss: 0.5368 - val_recall: 0.7396 Epoch 15/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5509 - recall: 0.7205 - val_loss: 0.5186 - val_recall: 0.7420 Epoch 16/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5478 - recall: 0.7267 - val_loss: 0.5284 - val_recall: 0.7641 Epoch 17/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5384 - recall: 0.7392 - val_loss: 0.5158 - val_recall: 0.7690 Epoch 18/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5382 - recall: 0.7329 - val_loss: 0.5072 - val_recall: 0.7543 Epoch 19/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5350 - recall: 0.7327 - val_loss: 0.5163 - val_recall: 0.7764 Epoch 20/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5301 - recall: 0.7459 - val_loss: 0.5057 - val_recall: 0.7518 Epoch 21/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5264 - recall: 0.7420 - val_loss: 0.5016 - val_recall: 0.7789 Epoch 22/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5235 - recall: 0.7550 - val_loss: 0.5136 - val_recall: 0.7887 Epoch 23/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5246 - recall: 0.7442 - val_loss: 0.4978 - val_recall: 0.7690 Epoch 24/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5181 - recall: 0.7430 - val_loss: 0.4960 - val_recall: 0.7666 Epoch 25/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5240 - recall: 0.7515 - val_loss: 0.5105 - val_recall: 0.8059 Epoch 26/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5139 - recall: 0.7613 - val_loss: 0.5018 - val_recall: 0.8108 Epoch 27/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5099 - recall: 0.7471 - val_loss: 0.5035 - val_recall: 0.7936 Epoch 28/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5132 - recall: 0.7586 - val_loss: 0.5054 - val_recall: 0.8059 Epoch 29/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5138 - recall: 0.7538 - val_loss: 0.5024 - val_recall: 0.7912 Epoch 30/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5065 - recall: 0.7623 - val_loss: 0.4942 - val_recall: 0.7764 Epoch 31/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5091 - recall: 0.7516 - val_loss: 0.4914 - val_recall: 0.7592 Epoch 32/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5091 - recall: 0.7583 - val_loss: 0.4863 - val_recall: 0.7789 Epoch 33/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5043 - recall: 0.7657 - val_loss: 0.4869 - val_recall: 0.7617 Epoch 34/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4994 - recall: 0.7532 - val_loss: 0.4919 - val_recall: 0.7592 Epoch 35/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5031 - recall: 0.7582 - val_loss: 0.4884 - val_recall: 0.7813 Epoch 36/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5006 - recall: 0.7559 - val_loss: 0.4807 - val_recall: 0.7617 Epoch 37/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4987 - recall: 0.7628 - val_loss: 0.4800 - val_recall: 0.7985 Epoch 38/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5025 - recall: 0.7629 - val_loss: 0.4768 - val_recall: 0.7862 Epoch 39/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4932 - recall: 0.7589 - val_loss: 0.4843 - val_recall: 0.7592 Epoch 40/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5006 - recall: 0.7640 - val_loss: 0.4771 - val_recall: 0.7445 Epoch 41/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4965 - recall: 0.7684 - val_loss: 0.4798 - val_recall: 0.8059 Epoch 42/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4973 - recall: 0.7644 - val_loss: 0.4882 - val_recall: 0.8034 Epoch 43/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4867 - recall: 0.7713 - val_loss: 0.4802 - val_recall: 0.7543 Epoch 44/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5004 - recall: 0.7565 - val_loss: 0.4834 - val_recall: 0.7469 Epoch 45/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4857 - recall: 0.7607 - val_loss: 0.4721 - val_recall: 0.7445 Epoch 46/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4868 - recall: 0.7642 - val_loss: 0.4810 - val_recall: 0.7494 Epoch 47/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4962 - recall: 0.7591 - val_loss: 0.4828 - val_recall: 0.7445 Epoch 48/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4950 - recall: 0.7679 - val_loss: 0.4767 - val_recall: 0.7322 Epoch 49/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4900 - recall: 0.7549 - val_loss: 0.4817 - val_recall: 0.7346 Epoch 50/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4887 - recall: 0.7631 - val_loss: 0.4792 - val_recall: 0.7297
Loss function
#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_dropout_smote_history001.history['loss'],adam_dropout_smote_history001.history['val_loss'])
- The loss in the validation and train are similar, though divergent, and shows a bit of noise in both sets, likely due to dropout and oversampling.
#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(adam_dropout_smote_history001.history['recall'],adam_dropout_smote_history001.history['val_recall'])
y_train_pred = adam_dropout_smote_model001.predict(X_train_smote)
#Predicting the results using 0.5 as the threshold
y_train_pred = (y_train_pred > 0.5)
y_train_pred
299/299 āāāāāāāāāāāāāāāāāāāā 0s 665us/step
array([[ True],
[ True],
[ True],
...,
[ True],
[False],
[ True]])
y_val_pred = adam_dropout_smote_model001.predict(X_val)
#Predicting the results using 0.5 as the threshold
y_val_pred = (y_val_pred > 0.5)
y_val_pred
63/63 āāāāāāāāāāāāāāāāāāāā 0s 547us/step
array([[False],
[False],
[ True],
...,
[False],
[False],
[ True]])
#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train_smote,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)
print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])
recall 0.793804 Name: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.001), dtype: float64 recall 0.72973 Name: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.001), dtype: float64
Classification report
print(classification_report(y_train_smote,y_train_pred))
precision recall f1-score support
0 0.80 0.81 0.80 4777
1 0.81 0.79 0.80 4777
accuracy 0.80 9554
macro avg 0.80 0.80 0.80 9554
weighted avg 0.80 0.80 0.80 9554
print(classification_report(y_val,y_val_pred))
precision recall f1-score support
0 0.92 0.80 0.86 1593
1 0.48 0.73 0.58 407
accuracy 0.78 2000
macro avg 0.70 0.76 0.72 2000
weighted avg 0.83 0.78 0.80 2000
Confusion matrix
#Calculating the confusion matrix
make_confusion_matrix(y_train_smote, y_train_pred)
#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)
Observations¶
- LR=0.001 didnt change the scores, lets try 0.002.
Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.002)¶
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)
model_name = "Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.002)"
#Initializing the model
adam_dropout_smote_model002 = Sequential()
adam_dropout_smote_model002.add(Dense(64,activation='relu',input_dim = X_train_smote.shape[1]))
adam_dropout_smote_model002.add(Dropout(0.8))
adam_dropout_smote_model002.add(Dense(32,activation='relu'))
adam_dropout_smote_model002.add(Dropout(0.4))
adam_dropout_smote_model002.add(Dense(16,activation='relu'))
adam_dropout_smote_model002.add(Dropout(0.2))
adam_dropout_smote_model002.add(Dense(8,activation='relu'))
adam_dropout_smote_model002.add(Dropout(0.1))
adam_dropout_smote_model002.add(Dense(1, activation = 'sigmoid'))
# use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam(learning_rate=0.002)
# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
# compile the model with binary cross entropy as loss function and recall as the metric
adam_dropout_smote_model002.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
adam_dropout_smote_model002.summary()
Model: "sequential"
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāā ā Layer (type) ā Output Shape ā Param # ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā dense (Dense) ā (None, 64) ā 768 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout (Dropout) ā (None, 64) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_1 (Dense) ā (None, 32) ā 2,080 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_1 (Dropout) ā (None, 32) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_2 (Dense) ā (None, 16) ā 528 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_2 (Dropout) ā (None, 16) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_3 (Dense) ā (None, 8) ā 136 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_3 (Dropout) ā (None, 8) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_4 (Dense) ā (None, 1) ā 9 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāā
Total params: 3,521 (13.75 KB)
Trainable params: 3,521 (13.75 KB)
Non-trainable params: 0 (0.00 B)
adam_dropout_smote_history002 = adam_dropout_smote_model002.fit(
X_train_smote,y_train_smote,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data = (X_val,y_val))
Epoch 1/50 150/150 āāāāāāāāāāāāāāāāāāāā 2s 3ms/step - loss: 0.6883 - recall: 0.5466 - val_loss: 0.6423 - val_recall: 0.6929 Epoch 2/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.6478 - recall: 0.6209 - val_loss: 0.6048 - val_recall: 0.7248 Epoch 3/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.6203 - recall: 0.6821 - val_loss: 0.5800 - val_recall: 0.7076 Epoch 4/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5970 - recall: 0.7129 - val_loss: 0.5853 - val_recall: 0.7396 Epoch 5/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5849 - recall: 0.7293 - val_loss: 0.5653 - val_recall: 0.7469 Epoch 6/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5741 - recall: 0.7282 - val_loss: 0.5781 - val_recall: 0.7690 Epoch 7/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5669 - recall: 0.7402 - val_loss: 0.5644 - val_recall: 0.7617 Epoch 8/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5667 - recall: 0.7326 - val_loss: 0.5533 - val_recall: 0.7715 Epoch 9/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5456 - recall: 0.7130 - val_loss: 0.5625 - val_recall: 0.7887 Epoch 10/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5429 - recall: 0.7310 - val_loss: 0.5244 - val_recall: 0.7764 Epoch 11/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5350 - recall: 0.7404 - val_loss: 0.5222 - val_recall: 0.7887 Epoch 12/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5297 - recall: 0.7494 - val_loss: 0.5327 - val_recall: 0.8059 Epoch 13/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5238 - recall: 0.7410 - val_loss: 0.5181 - val_recall: 0.7936 Epoch 14/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5245 - recall: 0.7411 - val_loss: 0.5160 - val_recall: 0.7936 Epoch 15/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5249 - recall: 0.7478 - val_loss: 0.5066 - val_recall: 0.8182 Epoch 16/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5231 - recall: 0.7499 - val_loss: 0.5050 - val_recall: 0.8084 Epoch 17/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5137 - recall: 0.7429 - val_loss: 0.5107 - val_recall: 0.8280 Epoch 18/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5197 - recall: 0.7403 - val_loss: 0.4949 - val_recall: 0.8108 Epoch 19/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5156 - recall: 0.7472 - val_loss: 0.5125 - val_recall: 0.8428 Epoch 20/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5100 - recall: 0.7709 - val_loss: 0.5149 - val_recall: 0.8182 Epoch 21/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5078 - recall: 0.7429 - val_loss: 0.5067 - val_recall: 0.8256 Epoch 22/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5057 - recall: 0.7483 - val_loss: 0.5061 - val_recall: 0.7715 Epoch 23/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - loss: 0.5097 - recall: 0.7399 - val_loss: 0.5030 - val_recall: 0.8182 Epoch 24/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5058 - recall: 0.7553 - val_loss: 0.5007 - val_recall: 0.8182 Epoch 25/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5080 - recall: 0.7596 - val_loss: 0.5038 - val_recall: 0.8378 Epoch 26/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4994 - recall: 0.7783 - val_loss: 0.5175 - val_recall: 0.8403 Epoch 27/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4980 - recall: 0.7580 - val_loss: 0.5124 - val_recall: 0.8256 Epoch 28/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5044 - recall: 0.7533 - val_loss: 0.4929 - val_recall: 0.7273 Epoch 29/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4958 - recall: 0.7649 - val_loss: 0.4925 - val_recall: 0.8256 Epoch 30/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4944 - recall: 0.7567 - val_loss: 0.4926 - val_recall: 0.8305 Epoch 31/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5021 - recall: 0.7611 - val_loss: 0.4967 - val_recall: 0.7224 Epoch 32/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4959 - recall: 0.7579 - val_loss: 0.4917 - val_recall: 0.8428 Epoch 33/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4973 - recall: 0.7707 - val_loss: 0.4785 - val_recall: 0.7101 Epoch 34/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4948 - recall: 0.7364 - val_loss: 0.4871 - val_recall: 0.8305 Epoch 35/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4995 - recall: 0.7599 - val_loss: 0.4933 - val_recall: 0.8256 Epoch 36/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4956 - recall: 0.7523 - val_loss: 0.4797 - val_recall: 0.7346 Epoch 37/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4881 - recall: 0.7717 - val_loss: 0.4985 - val_recall: 0.8673 Epoch 38/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4978 - recall: 0.7716 - val_loss: 0.4801 - val_recall: 0.8256 Epoch 39/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4894 - recall: 0.7636 - val_loss: 0.4834 - val_recall: 0.7322 Epoch 40/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4943 - recall: 0.7559 - val_loss: 0.4672 - val_recall: 0.7248 Epoch 41/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4962 - recall: 0.7495 - val_loss: 0.4885 - val_recall: 0.7445 Epoch 42/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4926 - recall: 0.7648 - val_loss: 0.4869 - val_recall: 0.7174 Epoch 43/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4857 - recall: 0.7455 - val_loss: 0.4805 - val_recall: 0.7224 Epoch 44/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4978 - recall: 0.7312 - val_loss: 0.4924 - val_recall: 0.7396 Epoch 45/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4850 - recall: 0.7508 - val_loss: 0.4687 - val_recall: 0.6658 Epoch 46/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4858 - recall: 0.7226 - val_loss: 0.4791 - val_recall: 0.7199 Epoch 47/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4978 - recall: 0.7585 - val_loss: 0.4862 - val_recall: 0.7101 Epoch 48/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4944 - recall: 0.7444 - val_loss: 0.4851 - val_recall: 0.7543 Epoch 49/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4913 - recall: 0.7448 - val_loss: 0.4807 - val_recall: 0.7371 Epoch 50/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4861 - recall: 0.7558 - val_loss: 0.4793 - val_recall: 0.7248
Loss function
#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_dropout_smote_history002.history['loss'],adam_dropout_smote_history002.history['val_loss'])
- The loss in the validation and train are similar, though divergent, and shows a bit of noise in both sets, likely due to dropout and oversampling.
#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(adam_dropout_smote_history002.history['recall'],adam_dropout_smote_history002.history['val_recall'])
y_train_pred = adam_dropout_smote_model002.predict(X_train_smote)
#Predicting the results using 0.5 as the threshold
y_train_pred = (y_train_pred > 0.5)
y_train_pred
299/299 āāāāāāāāāāāāāāāāāāāā 0s 659us/step
array([[ True],
[ True],
[ True],
...,
[ True],
[False],
[ True]])
y_val_pred = adam_dropout_smote_model002.predict(X_val)
#Predicting the results using 0.5 as the threshold
y_val_pred = (y_val_pred > 0.5)
y_val_pred
63/63 āāāāāāāāāāāāāāāāāāāā 0s 573us/step
array([[False],
[False],
[ True],
...,
[False],
[False],
[ True]])
#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train_smote,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)
print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])
recall 0.789617 Name: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.002), dtype: float64 recall 0.724816 Name: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.002), dtype: float64
Classification report
print(classification_report(y_train_smote,y_train_pred))
precision recall f1-score support
0 0.79 0.81 0.80 4777
1 0.81 0.79 0.80 4777
accuracy 0.80 9554
macro avg 0.80 0.80 0.80 9554
weighted avg 0.80 0.80 0.80 9554
print(classification_report(y_val,y_val_pred))
precision recall f1-score support
0 0.92 0.80 0.86 1593
1 0.48 0.72 0.58 407
accuracy 0.79 2000
macro avg 0.70 0.76 0.72 2000
weighted avg 0.83 0.79 0.80 2000
Confusion matrix
#Calculating the confusion matrix
make_confusion_matrix(y_train_smote, y_train_pred)
#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)
Observations¶
- 0.002 dropped the score a small amount.
- Lets try an order of magnitude shift.
Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01)¶
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)
model_name = "Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01)"
#Initializing the model
adam_dropout_smote_model01 = Sequential()
adam_dropout_smote_model01.add(Dense(64,activation='relu',input_dim = X_train_smote.shape[1]))
adam_dropout_smote_model01.add(Dropout(0.8))
adam_dropout_smote_model01.add(Dense(32,activation='relu'))
adam_dropout_smote_model01.add(Dropout(0.4))
adam_dropout_smote_model01.add(Dense(16,activation='relu'))
adam_dropout_smote_model01.add(Dropout(0.2))
adam_dropout_smote_model01.add(Dense(8,activation='relu'))
adam_dropout_smote_model01.add(Dropout(0.1))
adam_dropout_smote_model01.add(Dense(1, activation = 'sigmoid'))
# use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)
# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
# compile the model with binary cross entropy as loss function and recall as the metric
adam_dropout_smote_model01.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
adam_dropout_smote_model01.summary()
Model: "sequential"
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāā ā Layer (type) ā Output Shape ā Param # ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā dense (Dense) ā (None, 64) ā 768 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout (Dropout) ā (None, 64) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_1 (Dense) ā (None, 32) ā 2,080 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_1 (Dropout) ā (None, 32) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_2 (Dense) ā (None, 16) ā 528 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_2 (Dropout) ā (None, 16) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_3 (Dense) ā (None, 8) ā 136 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_3 (Dropout) ā (None, 8) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_4 (Dense) ā (None, 1) ā 9 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāā
Total params: 3,521 (13.75 KB)
Trainable params: 3,521 (13.75 KB)
Non-trainable params: 0 (0.00 B)
adam_dropout_smote_history01 = adam_dropout_smote_model01.fit(
X_train_smote,y_train_smote,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data = (X_val,y_val))
Epoch 1/50 150/150 āāāāāāāāāāāāāāāāāāāā 2s 3ms/step - loss: 0.6688 - recall: 0.5720 - val_loss: 0.5680 - val_recall: 0.7420 Epoch 2/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5985 - recall: 0.7138 - val_loss: 0.5430 - val_recall: 0.7469 Epoch 3/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5860 - recall: 0.7117 - val_loss: 0.5870 - val_recall: 0.8231 Epoch 4/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5708 - recall: 0.7045 - val_loss: 0.5388 - val_recall: 0.7641 Epoch 5/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5570 - recall: 0.7177 - val_loss: 0.5326 - val_recall: 0.7838 Epoch 6/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5527 - recall: 0.7159 - val_loss: 0.5367 - val_recall: 0.8354 Epoch 7/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5464 - recall: 0.7088 - val_loss: 0.5327 - val_recall: 0.7985 Epoch 8/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5472 - recall: 0.7244 - val_loss: 0.5078 - val_recall: 0.8034 Epoch 9/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5296 - recall: 0.7092 - val_loss: 0.4864 - val_recall: 0.8034 Epoch 10/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5339 - recall: 0.7199 - val_loss: 0.5272 - val_recall: 0.8231 Epoch 11/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5272 - recall: 0.7370 - val_loss: 0.5153 - val_recall: 0.8084 Epoch 12/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5347 - recall: 0.7104 - val_loss: 0.4895 - val_recall: 0.7961 Epoch 13/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5299 - recall: 0.7199 - val_loss: 0.5184 - val_recall: 0.7912 Epoch 14/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5236 - recall: 0.7191 - val_loss: 0.4812 - val_recall: 0.6929 Epoch 15/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5284 - recall: 0.6940 - val_loss: 0.5269 - val_recall: 0.8575 Epoch 16/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5333 - recall: 0.7324 - val_loss: 0.5162 - val_recall: 0.8600 Epoch 17/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5207 - recall: 0.7369 - val_loss: 0.4998 - val_recall: 0.8329 Epoch 18/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5202 - recall: 0.7340 - val_loss: 0.5021 - val_recall: 0.8452 Epoch 19/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5247 - recall: 0.7372 - val_loss: 0.5185 - val_recall: 0.8526 Epoch 20/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5203 - recall: 0.7481 - val_loss: 0.5085 - val_recall: 0.8084 Epoch 21/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5207 - recall: 0.7179 - val_loss: 0.5017 - val_recall: 0.8133 Epoch 22/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5200 - recall: 0.7123 - val_loss: 0.5096 - val_recall: 0.8157 Epoch 23/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5161 - recall: 0.7263 - val_loss: 0.5368 - val_recall: 0.8649 Epoch 24/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5159 - recall: 0.7363 - val_loss: 0.5174 - val_recall: 0.8771 Epoch 25/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5220 - recall: 0.7474 - val_loss: 0.5190 - val_recall: 0.8673 Epoch 26/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5162 - recall: 0.7433 - val_loss: 0.4944 - val_recall: 0.8575 Epoch 27/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5206 - recall: 0.7304 - val_loss: 0.5188 - val_recall: 0.8526 Epoch 28/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5177 - recall: 0.7389 - val_loss: 0.5136 - val_recall: 0.8526 Epoch 29/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5200 - recall: 0.7538 - val_loss: 0.4990 - val_recall: 0.8600 Epoch 30/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5147 - recall: 0.7293 - val_loss: 0.5106 - val_recall: 0.8550 Epoch 31/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5174 - recall: 0.7386 - val_loss: 0.5075 - val_recall: 0.8698 Epoch 32/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5149 - recall: 0.7374 - val_loss: 0.5000 - val_recall: 0.8649 Epoch 33/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5223 - recall: 0.7384 - val_loss: 0.4766 - val_recall: 0.7174 Epoch 34/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5195 - recall: 0.7322 - val_loss: 0.5064 - val_recall: 0.8870 Epoch 35/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5178 - recall: 0.7369 - val_loss: 0.4984 - val_recall: 0.8845 Epoch 36/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5145 - recall: 0.7388 - val_loss: 0.4840 - val_recall: 0.7273 Epoch 37/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5121 - recall: 0.7353 - val_loss: 0.4975 - val_recall: 0.8673 Epoch 38/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5217 - recall: 0.7424 - val_loss: 0.4545 - val_recall: 0.7666 Epoch 39/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5048 - recall: 0.7403 - val_loss: 0.4930 - val_recall: 0.7936 Epoch 40/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5184 - recall: 0.7034 - val_loss: 0.4640 - val_recall: 0.8231 Epoch 41/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5177 - recall: 0.7348 - val_loss: 0.4925 - val_recall: 0.8329 Epoch 42/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5206 - recall: 0.7208 - val_loss: 0.4611 - val_recall: 0.6609 Epoch 43/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5094 - recall: 0.7168 - val_loss: 0.4855 - val_recall: 0.8575 Epoch 44/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5186 - recall: 0.7289 - val_loss: 0.5084 - val_recall: 0.8403 Epoch 45/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5087 - recall: 0.7353 - val_loss: 0.4948 - val_recall: 0.8428 Epoch 46/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5182 - recall: 0.7438 - val_loss: 0.4959 - val_recall: 0.8354 Epoch 47/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5255 - recall: 0.7391 - val_loss: 0.5204 - val_recall: 0.8575 Epoch 48/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5213 - recall: 0.7382 - val_loss: 0.4954 - val_recall: 0.8796 Epoch 49/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5205 - recall: 0.7433 - val_loss: 0.5035 - val_recall: 0.8845 Epoch 50/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5049 - recall: 0.7420 - val_loss: 0.4801 - val_recall: 0.6781
Loss function
#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_dropout_smote_history01.history['loss'],adam_dropout_smote_history01.history['val_loss'])
- The loss in the validation and train are divergent, and shows a bit of noise in both sets though much more extreme in validation.
#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(adam_dropout_smote_history01.history['recall'],adam_dropout_smote_history01.history['val_recall'])
y_train_pred = adam_dropout_smote_model01.predict(X_train_smote)
#Predicting the results using 0.5 as the threshold
y_train_pred = (y_train_pred > 0.5)
y_train_pred
299/299 āāāāāāāāāāāāāāāāāāāā 0s 669us/step
array([[False],
[ True],
[ True],
...,
[ True],
[False],
[ True]])
y_val_pred = adam_dropout_smote_model01.predict(X_val)
#Predicting the results using 0.5 as the threshold
y_val_pred = (y_val_pred > 0.5)
y_val_pred
63/63 āāāāāāāāāāāāāāāāāāāā 0s 546us/step
array([[False],
[False],
[ True],
...,
[False],
[False],
[ True]])
#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train_smote,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)
print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])
recall 0.748796 Name: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01), dtype: float64 recall 0.678133 Name: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01), dtype: float64
Classification report
print(classification_report(y_train_smote,y_train_pred))
precision recall f1-score support
0 0.77 0.83 0.80 4777
1 0.82 0.75 0.78 4777
accuracy 0.79 9554
macro avg 0.79 0.79 0.79 9554
weighted avg 0.79 0.79 0.79 9554
print(classification_report(y_val,y_val_pred))
precision recall f1-score support
0 0.91 0.82 0.86 1593
1 0.49 0.68 0.57 407
accuracy 0.79 2000
macro avg 0.70 0.75 0.72 2000
weighted avg 0.82 0.79 0.80 2000
Confusion matrix
#Calculating the confusion matrix
make_confusion_matrix(y_train_smote, y_train_pred)
#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)
Observations¶
- That came out worse, lets try going back to adjusting on the thousandths.
Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.003)¶
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)
model_name = "Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.003)"
#Initializing the model
adam_dropout_smote_model = Sequential()
adam_dropout_smote_model.add(Dense(64,activation='relu',input_dim = X_train_smote.shape[1]))
adam_dropout_smote_model.add(Dropout(0.8))
adam_dropout_smote_model.add(Dense(32,activation='relu'))
adam_dropout_smote_model.add(Dropout(0.4))
adam_dropout_smote_model.add(Dense(16,activation='relu'))
adam_dropout_smote_model.add(Dropout(0.2))
adam_dropout_smote_model.add(Dense(8,activation='relu'))
adam_dropout_smote_model.add(Dropout(0.1))
adam_dropout_smote_model.add(Dense(1, activation = 'sigmoid'))
# use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam(learning_rate=0.003)
# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
# compile the model with binary cross entropy as loss function and recall as the metric
adam_dropout_smote_model.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
adam_dropout_smote_model.summary()
Model: "sequential"
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāā ā Layer (type) ā Output Shape ā Param # ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā dense (Dense) ā (None, 64) ā 768 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout (Dropout) ā (None, 64) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_1 (Dense) ā (None, 32) ā 2,080 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_1 (Dropout) ā (None, 32) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_2 (Dense) ā (None, 16) ā 528 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_2 (Dropout) ā (None, 16) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_3 (Dense) ā (None, 8) ā 136 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_3 (Dropout) ā (None, 8) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_4 (Dense) ā (None, 1) ā 9 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāā
Total params: 3,521 (13.75 KB)
Trainable params: 3,521 (13.75 KB)
Non-trainable params: 0 (0.00 B)
adam_dropout_smote_history = adam_dropout_smote_model.fit(
X_train_smote,y_train_smote,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data = (X_val,y_val))
Epoch 1/50 150/150 āāāāāāāāāāāāāāāāāāāā 2s 3ms/step - loss: 0.6833 - recall: 0.5378 - val_loss: 0.6247 - val_recall: 0.7273 Epoch 2/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.6334 - recall: 0.6912 - val_loss: 0.5943 - val_recall: 0.7297 Epoch 3/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.6053 - recall: 0.7153 - val_loss: 0.5781 - val_recall: 0.7420 Epoch 4/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5816 - recall: 0.7239 - val_loss: 0.5674 - val_recall: 0.7371 Epoch 5/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5683 - recall: 0.7251 - val_loss: 0.5552 - val_recall: 0.7543 Epoch 6/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5585 - recall: 0.7175 - val_loss: 0.5371 - val_recall: 0.7641 Epoch 7/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5524 - recall: 0.7342 - val_loss: 0.5581 - val_recall: 0.8108 Epoch 8/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5535 - recall: 0.7260 - val_loss: 0.5361 - val_recall: 0.7887 Epoch 9/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5294 - recall: 0.7201 - val_loss: 0.5310 - val_recall: 0.7887 Epoch 10/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5331 - recall: 0.7147 - val_loss: 0.5384 - val_recall: 0.8329 Epoch 11/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5240 - recall: 0.7486 - val_loss: 0.5470 - val_recall: 0.8354 Epoch 12/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5221 - recall: 0.7518 - val_loss: 0.5166 - val_recall: 0.8133 Epoch 13/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5150 - recall: 0.7446 - val_loss: 0.5214 - val_recall: 0.8157 Epoch 14/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5186 - recall: 0.7391 - val_loss: 0.5223 - val_recall: 0.8182 Epoch 15/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5176 - recall: 0.7462 - val_loss: 0.5118 - val_recall: 0.8182 Epoch 16/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5203 - recall: 0.7498 - val_loss: 0.5106 - val_recall: 0.8182 Epoch 17/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5098 - recall: 0.7542 - val_loss: 0.4984 - val_recall: 0.8280 Epoch 18/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5124 - recall: 0.7467 - val_loss: 0.4853 - val_recall: 0.8010 Epoch 19/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5130 - recall: 0.7382 - val_loss: 0.5020 - val_recall: 0.8206 Epoch 20/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5049 - recall: 0.7645 - val_loss: 0.5144 - val_recall: 0.8231 Epoch 21/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5018 - recall: 0.7590 - val_loss: 0.4982 - val_recall: 0.8182 Epoch 22/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5038 - recall: 0.7436 - val_loss: 0.4950 - val_recall: 0.8403 Epoch 23/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5035 - recall: 0.7754 - val_loss: 0.5201 - val_recall: 0.8550 Epoch 24/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5027 - recall: 0.7652 - val_loss: 0.4998 - val_recall: 0.8403 Epoch 25/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5078 - recall: 0.7588 - val_loss: 0.5085 - val_recall: 0.8600 Epoch 26/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5024 - recall: 0.7637 - val_loss: 0.5181 - val_recall: 0.8600 Epoch 27/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4987 - recall: 0.7631 - val_loss: 0.5161 - val_recall: 0.8354 Epoch 28/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5030 - recall: 0.7461 - val_loss: 0.4991 - val_recall: 0.8256 Epoch 29/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5038 - recall: 0.7578 - val_loss: 0.4793 - val_recall: 0.7617 Epoch 30/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4999 - recall: 0.7659 - val_loss: 0.4939 - val_recall: 0.7764 Epoch 31/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4985 - recall: 0.7629 - val_loss: 0.4921 - val_recall: 0.7813 Epoch 32/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4950 - recall: 0.7712 - val_loss: 0.4744 - val_recall: 0.7224 Epoch 33/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4972 - recall: 0.7581 - val_loss: 0.4839 - val_recall: 0.7248 Epoch 34/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5002 - recall: 0.7354 - val_loss: 0.4717 - val_recall: 0.7248 Epoch 35/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5019 - recall: 0.7348 - val_loss: 0.5001 - val_recall: 0.7985 Epoch 36/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4976 - recall: 0.7463 - val_loss: 0.4871 - val_recall: 0.7518 Epoch 37/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4934 - recall: 0.7464 - val_loss: 0.4988 - val_recall: 0.7641 Epoch 38/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4954 - recall: 0.7546 - val_loss: 0.4830 - val_recall: 0.7273 Epoch 39/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4909 - recall: 0.7411 - val_loss: 0.4940 - val_recall: 0.7371 Epoch 40/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4965 - recall: 0.7507 - val_loss: 0.4887 - val_recall: 0.7641 Epoch 41/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4962 - recall: 0.7386 - val_loss: 0.4767 - val_recall: 0.7027 Epoch 42/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4989 - recall: 0.7138 - val_loss: 0.4902 - val_recall: 0.7224 Epoch 43/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4910 - recall: 0.7375 - val_loss: 0.4813 - val_recall: 0.7273 Epoch 44/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4962 - recall: 0.7446 - val_loss: 0.4853 - val_recall: 0.7715 Epoch 45/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4903 - recall: 0.7446 - val_loss: 0.4891 - val_recall: 0.7445 Epoch 46/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4901 - recall: 0.7399 - val_loss: 0.4702 - val_recall: 0.7076 Epoch 47/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4982 - recall: 0.7241 - val_loss: 0.4849 - val_recall: 0.7396 Epoch 48/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4950 - recall: 0.7329 - val_loss: 0.4865 - val_recall: 0.7125 Epoch 49/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4962 - recall: 0.7098 - val_loss: 0.4653 - val_recall: 0.8010 Epoch 50/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.4854 - recall: 0.7496 - val_loss: 0.4865 - val_recall: 0.7961
Loss function
#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_dropout_smote_history.history['loss'],adam_dropout_smote_history.history['val_loss'])
- The loss in the validation and train are similar, though divergent, and shows a bit of noise in both sets though much more extreme in validation.
#Plotting Train recall vs Validation recall
#plot_train_vs_val_recall(adam_dropout_smote_history.history['recall'],adam_dropout_smote_history.history['val_recall'])
y_train_pred = adam_dropout_smote_model.predict(X_train_smote)
#Predicting the results using 0.5 as the threshold
y_train_pred = (y_train_pred > 0.5)
y_train_pred
299/299 āāāāāāāāāāāāāāāāāāāā 0s 665us/step
array([[ True],
[ True],
[ True],
...,
[ True],
[False],
[ True]])
y_val_pred = adam_dropout_smote_model.predict(X_val)
#Predicting the results using 0.5 as the threshold
y_val_pred = (y_val_pred > 0.5)
y_val_pred
63/63 āāāāāāāāāāāāāāāāāāāā 0s 564us/step
array([[False],
[False],
[ True],
...,
[False],
[False],
[ True]])
#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train_smote,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)
print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])
recall 0.83609 Name: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.003), dtype: float64 recall 0.796069 Name: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.003), dtype: float64
Classification report
print(classification_report(y_train_smote,y_train_pred))
precision recall f1-score support
0 0.82 0.75 0.78 4777
1 0.77 0.84 0.80 4777
accuracy 0.79 9554
macro avg 0.80 0.79 0.79 9554
weighted avg 0.80 0.79 0.79 9554
print(classification_report(y_val,y_val_pred))
precision recall f1-score support
0 0.94 0.75 0.83 1593
1 0.45 0.80 0.57 407
accuracy 0.76 2000
macro avg 0.69 0.77 0.70 2000
weighted avg 0.84 0.76 0.78 2000
Confusion matrix
#Calculating the confusion matrix
make_confusion_matrix(y_train_smote, y_train_pred)
#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)
Observations¶
- Ok, that moved the bar pretty good. 84% on training and 80 on validation is pretty good.
- Lets see if adjusting batch size helps improve things.
Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, Dropout, Adjusting LR and Batch size¶
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)
model_name = "Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32)"
#Initializing the model
adam_dropout_smote_model2 = Sequential()
adam_dropout_smote_model2.add(Dense(64,activation='relu',input_dim = X_train_smote.shape[1]))
adam_dropout_smote_model2.add(Dropout(0.8))
adam_dropout_smote_model2.add(Dense(32,activation='relu'))
adam_dropout_smote_model2.add(Dropout(0.4))
adam_dropout_smote_model2.add(Dense(16,activation='relu'))
adam_dropout_smote_model2.add(Dropout(0.2))
adam_dropout_smote_model2.add(Dense(8,activation='relu'))
adam_dropout_smote_model2.add(Dropout(0.1))
adam_dropout_smote_model2.add(Dense(1, activation = 'sigmoid'))
# use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)
# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
# compile the model with binary cross entropy as loss function and recall as the metric
adam_dropout_smote_model2.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
adam_dropout_smote_model2.summary()
Model: "sequential"
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāā ā Layer (type) ā Output Shape ā Param # ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā dense (Dense) ā (None, 64) ā 768 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout (Dropout) ā (None, 64) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_1 (Dense) ā (None, 32) ā 2,080 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_1 (Dropout) ā (None, 32) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_2 (Dense) ā (None, 16) ā 528 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_2 (Dropout) ā (None, 16) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_3 (Dense) ā (None, 8) ā 136 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_3 (Dropout) ā (None, 8) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_4 (Dense) ā (None, 1) ā 9 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāā
Total params: 3,521 (13.75 KB)
Trainable params: 3,521 (13.75 KB)
Non-trainable params: 0 (0.00 B)
adam_dropout_smote_history2 = adam_dropout_smote_model2.fit(
X_train_smote,y_train_smote,
batch_size=32,
epochs=epochs,
verbose=1,
validation_data = (X_val,y_val))
Epoch 1/50 299/299 āāāāāāāāāāāāāāāāāāāā 2s 2ms/step - loss: 0.6685 - recall: 0.5311 - val_loss: 0.5719 - val_recall: 0.7346 Epoch 2/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.6080 - recall: 0.7243 - val_loss: 0.5628 - val_recall: 0.7322 Epoch 3/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5825 - recall: 0.6946 - val_loss: 0.5659 - val_recall: 0.7248 Epoch 4/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5829 - recall: 0.7207 - val_loss: 0.5347 - val_recall: 0.7052 Epoch 5/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5738 - recall: 0.7017 - val_loss: 0.5653 - val_recall: 0.7297 Epoch 6/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5572 - recall: 0.7008 - val_loss: 0.5469 - val_recall: 0.8084 Epoch 7/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5646 - recall: 0.7199 - val_loss: 0.5105 - val_recall: 0.7838 Epoch 8/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5562 - recall: 0.7194 - val_loss: 0.5553 - val_recall: 0.7690 Epoch 9/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5507 - recall: 0.7286 - val_loss: 0.5179 - val_recall: 0.8305 Epoch 10/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5487 - recall: 0.7397 - val_loss: 0.5314 - val_recall: 0.8280 Epoch 11/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5452 - recall: 0.7281 - val_loss: 0.5365 - val_recall: 0.8550 Epoch 12/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5520 - recall: 0.7302 - val_loss: 0.5733 - val_recall: 0.8600 Epoch 13/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5476 - recall: 0.7354 - val_loss: 0.5127 - val_recall: 0.8501 Epoch 14/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5470 - recall: 0.7262 - val_loss: 0.5255 - val_recall: 0.8329 Epoch 15/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5364 - recall: 0.7385 - val_loss: 0.5772 - val_recall: 0.8256 Epoch 16/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5401 - recall: 0.7251 - val_loss: 0.5301 - val_recall: 0.8256 Epoch 17/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5404 - recall: 0.7420 - val_loss: 0.5593 - val_recall: 0.8624 Epoch 18/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5412 - recall: 0.7246 - val_loss: 0.5316 - val_recall: 0.8280 Epoch 19/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5264 - recall: 0.7239 - val_loss: 0.5205 - val_recall: 0.7813 Epoch 20/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5313 - recall: 0.7253 - val_loss: 0.5273 - val_recall: 0.8305 Epoch 21/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5441 - recall: 0.7189 - val_loss: 0.5437 - val_recall: 0.8378 Epoch 22/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5417 - recall: 0.7300 - val_loss: 0.5310 - val_recall: 0.8206 Epoch 23/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5353 - recall: 0.7455 - val_loss: 0.5285 - val_recall: 0.8477 Epoch 24/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5328 - recall: 0.7456 - val_loss: 0.5232 - val_recall: 0.8280 Epoch 25/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5410 - recall: 0.7349 - val_loss: 0.4805 - val_recall: 0.8206 Epoch 26/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5309 - recall: 0.7461 - val_loss: 0.5054 - val_recall: 0.8157 Epoch 27/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5391 - recall: 0.7507 - val_loss: 0.5353 - val_recall: 0.8354 Epoch 28/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5287 - recall: 0.7307 - val_loss: 0.5272 - val_recall: 0.8378 Epoch 29/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5369 - recall: 0.7214 - val_loss: 0.4839 - val_recall: 0.8010 Epoch 30/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5273 - recall: 0.7123 - val_loss: 0.5522 - val_recall: 0.8059 Epoch 31/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5332 - recall: 0.6987 - val_loss: 0.5560 - val_recall: 0.8059 Epoch 32/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5286 - recall: 0.7385 - val_loss: 0.5449 - val_recall: 0.7740 Epoch 33/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5337 - recall: 0.7113 - val_loss: 0.4915 - val_recall: 0.7740 Epoch 34/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5424 - recall: 0.7431 - val_loss: 0.5259 - val_recall: 0.8477 Epoch 35/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5331 - recall: 0.7499 - val_loss: 0.5055 - val_recall: 0.8133 Epoch 36/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5244 - recall: 0.7161 - val_loss: 0.5418 - val_recall: 0.8280 Epoch 37/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5326 - recall: 0.7395 - val_loss: 0.5211 - val_recall: 0.8256 Epoch 38/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5295 - recall: 0.7285 - val_loss: 0.5156 - val_recall: 0.8157 Epoch 39/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5317 - recall: 0.7277 - val_loss: 0.5169 - val_recall: 0.8231 Epoch 40/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5266 - recall: 0.7195 - val_loss: 0.5070 - val_recall: 0.8452 Epoch 41/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5246 - recall: 0.7496 - val_loss: 0.5370 - val_recall: 0.8354 Epoch 42/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5295 - recall: 0.7173 - val_loss: 0.5322 - val_recall: 0.8403 Epoch 43/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5383 - recall: 0.7372 - val_loss: 0.5143 - val_recall: 0.8133 Epoch 44/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5254 - recall: 0.7311 - val_loss: 0.5291 - val_recall: 0.8378 Epoch 45/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5266 - recall: 0.7398 - val_loss: 0.5418 - val_recall: 0.8305 Epoch 46/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5295 - recall: 0.7382 - val_loss: 0.5059 - val_recall: 0.8157 Epoch 47/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5273 - recall: 0.7067 - val_loss: 0.5265 - val_recall: 0.8305 Epoch 48/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5302 - recall: 0.7268 - val_loss: 0.5078 - val_recall: 0.8305 Epoch 49/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5420 - recall: 0.7211 - val_loss: 0.4950 - val_recall: 0.8010 Epoch 50/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5367 - recall: 0.7291 - val_loss: 0.5010 - val_recall: 0.8305
Loss function
#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_dropout_smote_history2.history['loss'],adam_dropout_smote_history2.history['val_loss'])
- The loss in the validation and train are divergent, and shows a bit of noise in both sets though much more extreme in validation.
- We can see right around 22 or 23 the train and validation set are very similar and close. We should try exiting around that number and see how it improves things.
y_train_pred = adam_dropout_smote_model2.predict(X_train_smote)
#Predicting the results using 0.5 as the threshold
y_train_pred = (y_train_pred > 0.5)
y_train_pred
299/299 āāāāāāāāāāāāāāāāāāāā 0s 663us/step
array([[ True],
[ True],
[ True],
...,
[ True],
[False],
[ True]])
y_val_pred = adam_dropout_smote_model2.predict(X_val)
#Predicting the results using 0.5 as the threshold
y_val_pred = (y_val_pred > 0.5)
y_val_pred
63/63 āāāāāāāāāāāāāāāāāāāā 0s 552us/step
array([[False],
[False],
[ True],
...,
[False],
[False],
[ True]])
#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train_smote,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)
print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])
recall 0.868327 Name: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32), dtype: float64 recall 0.830467 Name: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32), dtype: float64
Classification report
print(classification_report(y_train_smote,y_train_pred))
precision recall f1-score support
0 0.84 0.70 0.76 4777
1 0.74 0.87 0.80 4777
accuracy 0.78 9554
macro avg 0.79 0.78 0.78 9554
weighted avg 0.79 0.78 0.78 9554
print(classification_report(y_val,y_val_pred))
precision recall f1-score support
0 0.94 0.69 0.80 1593
1 0.41 0.83 0.55 407
accuracy 0.72 2000
macro avg 0.67 0.76 0.67 2000
weighted avg 0.83 0.72 0.75 2000
Confusion matrix
#Calculating the confusion matrix
make_confusion_matrix(y_train_smote, y_train_pred)
#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)
Observations¶
- That showed some further improvement up to 87/83.
- Lets see if adjusting the epochs will help.
Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32) + Epochs 23¶
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)
model_name = "Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32) + Epochs 23"
#Initializing the model
adam_dropout_smote_model2c = Sequential()
adam_dropout_smote_model2c.add(Dense(64,activation='relu',input_dim = X_train_smote.shape[1]))
adam_dropout_smote_model2c.add(Dropout(0.8))
adam_dropout_smote_model2c.add(Dense(32,activation='relu'))
adam_dropout_smote_model2c.add(Dropout(0.4))
adam_dropout_smote_model2c.add(Dense(16,activation='relu'))
adam_dropout_smote_model2c.add(Dropout(0.2))
adam_dropout_smote_model2c.add(Dense(8,activation='relu'))
adam_dropout_smote_model2c.add(Dropout(0.1))
adam_dropout_smote_model2c.add(Dense(1, activation = 'sigmoid'))
# use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)
# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
# compile the model with binary cross entropy as loss function and recall as the metric
adam_dropout_smote_model2c.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
adam_dropout_smote_model2c.summary()
Model: "sequential"
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāā ā Layer (type) ā Output Shape ā Param # ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā dense (Dense) ā (None, 64) ā 768 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout (Dropout) ā (None, 64) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_1 (Dense) ā (None, 32) ā 2,080 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_1 (Dropout) ā (None, 32) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_2 (Dense) ā (None, 16) ā 528 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_2 (Dropout) ā (None, 16) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_3 (Dense) ā (None, 8) ā 136 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_3 (Dropout) ā (None, 8) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_4 (Dense) ā (None, 1) ā 9 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāā
Total params: 3,521 (13.75 KB)
Trainable params: 3,521 (13.75 KB)
Non-trainable params: 0 (0.00 B)
adam_dropout_smote_history2c = adam_dropout_smote_model2c.fit(
X_train_smote,y_train_smote,
batch_size=32,
epochs=23,
verbose=1,
validation_data = (X_val,y_val))
Epoch 1/23 299/299 āāāāāāāāāāāāāāāāāāāā 2s 2ms/step - loss: 0.6685 - recall: 0.5311 - val_loss: 0.5719 - val_recall: 0.7346 Epoch 2/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.6080 - recall: 0.7243 - val_loss: 0.5628 - val_recall: 0.7322 Epoch 3/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5825 - recall: 0.6946 - val_loss: 0.5659 - val_recall: 0.7248 Epoch 4/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5829 - recall: 0.7207 - val_loss: 0.5347 - val_recall: 0.7052 Epoch 5/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5738 - recall: 0.7017 - val_loss: 0.5653 - val_recall: 0.7297 Epoch 6/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5572 - recall: 0.7008 - val_loss: 0.5469 - val_recall: 0.8084 Epoch 7/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5646 - recall: 0.7199 - val_loss: 0.5105 - val_recall: 0.7838 Epoch 8/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5562 - recall: 0.7194 - val_loss: 0.5553 - val_recall: 0.7690 Epoch 9/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5507 - recall: 0.7286 - val_loss: 0.5179 - val_recall: 0.8305 Epoch 10/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5487 - recall: 0.7397 - val_loss: 0.5314 - val_recall: 0.8280 Epoch 11/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5452 - recall: 0.7281 - val_loss: 0.5365 - val_recall: 0.8550 Epoch 12/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5520 - recall: 0.7302 - val_loss: 0.5733 - val_recall: 0.8600 Epoch 13/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5476 - recall: 0.7354 - val_loss: 0.5127 - val_recall: 0.8501 Epoch 14/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5470 - recall: 0.7262 - val_loss: 0.5255 - val_recall: 0.8329 Epoch 15/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5364 - recall: 0.7385 - val_loss: 0.5772 - val_recall: 0.8256 Epoch 16/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5401 - recall: 0.7251 - val_loss: 0.5301 - val_recall: 0.8256 Epoch 17/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5404 - recall: 0.7420 - val_loss: 0.5593 - val_recall: 0.8624 Epoch 18/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5412 - recall: 0.7246 - val_loss: 0.5316 - val_recall: 0.8280 Epoch 19/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5264 - recall: 0.7239 - val_loss: 0.5205 - val_recall: 0.7813 Epoch 20/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5313 - recall: 0.7253 - val_loss: 0.5273 - val_recall: 0.8305 Epoch 21/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5441 - recall: 0.7189 - val_loss: 0.5437 - val_recall: 0.8378 Epoch 22/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5417 - recall: 0.7300 - val_loss: 0.5310 - val_recall: 0.8206 Epoch 23/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5353 - recall: 0.7455 - val_loss: 0.5285 - val_recall: 0.8477
Loss function
#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_dropout_smote_history2c.history['loss'],adam_dropout_smote_history2c.history['val_loss'])
- The loss in the validation and train are divergent, and shows a bit of noise in both sets though much more extreme in validation. Convergence is close in the final epochs.
y_train_pred = adam_dropout_smote_model2c.predict(X_train_smote)
#Predicting the results using 0.5 as the threshold
y_train_pred = (y_train_pred > 0.5)
y_train_pred
299/299 āāāāāāāāāāāāāāāāāāāā 0s 664us/step
array([[ True],
[ True],
[ True],
...,
[ True],
[False],
[ True]])
y_val_pred = adam_dropout_smote_model2c.predict(X_val)
#Predicting the results using 0.5 as the threshold
y_val_pred = (y_val_pred > 0.5)
y_val_pred
63/63 āāāāāāāāāāāāāāāāāāāā 0s 544us/step
array([[False],
[False],
[ True],
...,
[False],
[False],
[ True]])
#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train_smote,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)
print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])
recall 0.864559 Name: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32) + Epochs 23, dtype: float64 recall 0.847666 Name: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32) + Epochs 23, dtype: float64
Classification report
print(classification_report(y_train_smote,y_train_pred))
precision recall f1-score support
0 0.84 0.69 0.76 4777
1 0.74 0.86 0.80 4777
accuracy 0.78 9554
macro avg 0.79 0.78 0.78 9554
weighted avg 0.79 0.78 0.78 9554
print(classification_report(y_val,y_val_pred))
precision recall f1-score support
0 0.95 0.69 0.80 1593
1 0.41 0.85 0.55 407
accuracy 0.72 2000
macro avg 0.68 0.77 0.67 2000
weighted avg 0.84 0.72 0.75 2000
Confusion matrix
#Calculating the confusion matrix
make_confusion_matrix(y_train_smote, y_train_pred)
#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)
Observations¶
- Training set dropped a little bit, but validation came up by almost 2 full points.
- Train and validation sets also have the least difference of any other model.
- This will be our final model.
Model Performance Comparison and Final Model Selection¶
print("Training performance comparison")
training_metrics
Training performance comparison
| recall | |
|---|---|
| Neural Network (64,32,16,8) with SGD | 0.429272 |
| Neural Network (64,32,16,8) with SGD + Momentum (0.9) | 0.518397 |
| Neural Network (64,32,16,8) with Adam | 0.651676 |
| Neural Network (256,128,64,32,16,8) with Adam | 0.806214 |
| Neural Network (256,128,64,128,64,8) with Adam + LR (0.001) | 0.876533 |
| Neural Network (64,32,16,8) with Adam & Dropout (0.4, 0.3, 0.2, 0.1) | 0.510221 |
| Neural Network (32,16,8) with Adam & Dropout (0.4, 0.3, 0.2) | 0.420278 |
| Neural Network (64,32) with Adam & Dropout (0.5, 0.5) | 0.348324 |
| Neural Network (64,32,16,8) with Adam, He Normal & Dropout (0.4, 0.3, 0.2, 0.1) | 0.376124 |
| Neural Network (64,32,16,8) with Adam & Batch Normalization | 0.683565 |
| Neural Network (64,32,16,8) with Adam, Batch Normalization & Dropout (0.4, 0.3, 0.2, 0.1) | 0.413737 |
| Neural Network (64,32,16,8) with SMOTE & SGD | 0.805526 |
| Neural Network (64,32,16,8) with SMOTE & Adam | 0.941386 |
| Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) | 0.793804 |
| Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.001) | 0.793804 |
| Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.002) | 0.789617 |
| Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) | 0.748796 |
| Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.003) | 0.836090 |
| Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32) | 0.868327 |
| Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32) + Epochs 23 | 0.864559 |
print("Validation set performance comparison")
validation_metrics
Validation set performance comparison
| recall | |
|---|---|
| Neural Network (64,32,16,8) with SGD | 0.398034 |
| Neural Network (64,32,16,8) with SGD + Momentum (0.9) | 0.405405 |
| Neural Network (64,32,16,8) with Adam | 0.479115 |
| Neural Network (256,128,64,32,16,8) with Adam | 0.491400 |
| Neural Network (256,128,64,128,64,8) with Adam + LR (0.001) | 0.457002 |
| Neural Network (64,32,16,8) with Adam & Dropout (0.4, 0.3, 0.2, 0.1) | 0.474201 |
| Neural Network (32,16,8) with Adam & Dropout (0.4, 0.3, 0.2) | 0.383292 |
| Neural Network (64,32) with Adam & Dropout (0.5, 0.5) | 0.304668 |
| Neural Network (64,32,16,8) with Adam, He Normal & Dropout (0.4, 0.3, 0.2, 0.1) | 0.331695 |
| Neural Network (64,32,16,8) with Adam & Batch Normalization | 0.461916 |
| Neural Network (64,32,16,8) with Adam, Batch Normalization & Dropout (0.4, 0.3, 0.2, 0.1) | 0.366093 |
| Neural Network (64,32,16,8) with SMOTE & SGD | 0.727273 |
| Neural Network (64,32,16,8) with SMOTE & Adam | 0.651106 |
| Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) | 0.729730 |
| Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.001) | 0.729730 |
| Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.002) | 0.724816 |
| Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) | 0.678133 |
| Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.003) | 0.796069 |
| Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32) | 0.830467 |
| Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32) + Epochs 23 | 0.847666 |
# Compare diffs
training_metrics - validation_metrics
| recall | |
|---|---|
| Neural Network (64,32,16,8) with SGD | 0.031238 |
| Neural Network (64,32,16,8) with SGD + Momentum (0.9) | 0.112992 |
| Neural Network (64,32,16,8) with Adam | 0.172561 |
| Neural Network (256,128,64,32,16,8) with Adam | 0.314814 |
| Neural Network (256,128,64,128,64,8) with Adam + LR (0.001) | 0.419531 |
| Neural Network (64,32,16,8) with Adam & Dropout (0.4, 0.3, 0.2, 0.1) | 0.036019 |
| Neural Network (32,16,8) with Adam & Dropout (0.4, 0.3, 0.2) | 0.036986 |
| Neural Network (64,32) with Adam & Dropout (0.5, 0.5) | 0.043655 |
| Neural Network (64,32,16,8) with Adam, He Normal & Dropout (0.4, 0.3, 0.2, 0.1) | 0.044429 |
| Neural Network (64,32,16,8) with Adam & Batch Normalization | 0.221649 |
| Neural Network (64,32,16,8) with Adam, Batch Normalization & Dropout (0.4, 0.3, 0.2, 0.1) | 0.047643 |
| Neural Network (64,32,16,8) with SMOTE & SGD | 0.078254 |
| Neural Network (64,32,16,8) with SMOTE & Adam | 0.290280 |
| Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) | 0.064074 |
| Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.001) | 0.064074 |
| Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.002) | 0.064801 |
| Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) | 0.070664 |
| Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.003) | 0.040021 |
| Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32) | 0.037861 |
| Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32) + Epochs 23 | 0.016893 |
Observations¶
Training data¶
- Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32) at 86.8%
- Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32) + Epochs 23 at 86.4%
Validation data¶
- While epochs 50 version performs better in training, the epochs 23 version beats it in validation set at 84.7% to 83%.
- Epochs 23 version also is much closer in the 2 values at 1.7% difference vs 3.8% difference.
Conclusion¶
- For this reason, the Epochs 23 version is our final model selection.
Final Model¶
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)
model = "Final Model: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32) + Epochs 23"
#Initializing the model
final_test_model = Sequential()
final_test_model.add(Dense(64,activation='relu',input_dim = X_train_smote.shape[1]))
final_test_model.add(Dropout(0.8))
final_test_model.add(Dense(32,activation='relu'))
final_test_model.add(Dropout(0.4))
final_test_model.add(Dense(16,activation='relu'))
final_test_model.add(Dropout(0.2))
final_test_model.add(Dense(8,activation='relu'))
final_test_model.add(Dropout(0.1))
final_test_model.add(Dense(1, activation = 'sigmoid'))
#use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)
# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
# compile the model with binary cross entropy as loss function and recall as the metric
final_test_model.compile(loss='binary_crossentropy',optimizer=optimizer)
final_test_model.summary()
Model: "sequential"
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāā ā Layer (type) ā Output Shape ā Param # ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā dense (Dense) ā (None, 64) ā 768 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout (Dropout) ā (None, 64) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_1 (Dense) ā (None, 32) ā 2,080 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_1 (Dropout) ā (None, 32) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_2 (Dense) ā (None, 16) ā 528 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_2 (Dropout) ā (None, 16) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_3 (Dense) ā (None, 8) ā 136 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_3 (Dropout) ā (None, 8) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_4 (Dense) ā (None, 1) ā 9 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāā
Total params: 3,521 (13.75 KB)
Trainable params: 3,521 (13.75 KB)
Non-trainable params: 0 (0.00 B)
final_test_history = final_test_model.fit(
X_train_smote,y_train_smote,
batch_size=32,
epochs=23,
verbose=1,
validation_data = (X_val,y_val))
Epoch 1/23 299/299 āāāāāāāāāāāāāāāāāāāā 2s 2ms/step - loss: 0.6685 - val_loss: 0.5719 Epoch 2/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.6080 - val_loss: 0.5628 Epoch 3/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5825 - val_loss: 0.5659 Epoch 4/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5829 - val_loss: 0.5347 Epoch 5/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5738 - val_loss: 0.5653 Epoch 6/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5572 - val_loss: 0.5469 Epoch 7/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5646 - val_loss: 0.5105 Epoch 8/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5562 - val_loss: 0.5553 Epoch 9/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5507 - val_loss: 0.5179 Epoch 10/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5487 - val_loss: 0.5314 Epoch 11/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5452 - val_loss: 0.5365 Epoch 12/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5520 - val_loss: 0.5733 Epoch 13/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5476 - val_loss: 0.5127 Epoch 14/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5470 - val_loss: 0.5255 Epoch 15/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5364 - val_loss: 0.5772 Epoch 16/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5401 - val_loss: 0.5301 Epoch 17/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5404 - val_loss: 0.5593 Epoch 18/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5412 - val_loss: 0.5316 Epoch 19/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5264 - val_loss: 0.5205 Epoch 20/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5313 - val_loss: 0.5273 Epoch 21/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5441 - val_loss: 0.5437 Epoch 22/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5417 - val_loss: 0.5310 Epoch 23/23 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5353 - val_loss: 0.5285
Loss function
Classification report
y_test_pred = final_test_model.predict(X_test)
y_test_pred = (y_test_pred > 0.5)
print(y_test_pred)
63/63 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step [[False] [False] [False] ... [ True] [False] [False]]
#lets print classification report
cr=classification_report(y_test,y_test_pred)
print(cr)
precision recall f1-score support
0 0.94 0.67 0.78 1593
1 0.39 0.84 0.53 407
accuracy 0.70 2000
macro avg 0.67 0.75 0.66 2000
weighted avg 0.83 0.70 0.73 2000
Confusion matrix
#Calculating the confusion matrix
make_confusion_matrix(y_test,y_test_pred)
Actionable Insights and Business Recommendations¶
Observations¶
- The majority of model gave recall between 50% to 80% with a few showing low recall in the 30-40% range.
- The data set was imbalanced, and had we had a better data set we might have improved performance even more.
- By using oversampling we were able to improve the models significantly, but there was a lot of noise present in those models.
Conclusion¶
- The best performing model is "Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.1) + LR (0.01) + Batch (32) + Epochs 23", where we used oversampling, a learning rate of 0.01 with the Adam optimizer, used aggressive dropout on each layer, and limited the epochs to 23. While very noisy, This model gave us a recall of 83.5% on the test data set and a difference of 1.7% between the Training and Validation sets.
- By using this model, the bank can identify customers who are willing to leave the bank and enact an action plan to retain these customers.
Business Recommendations¶
In our analysis, we found that customers who have less than 1 year or more than 9 of Tenure are more likely to churn. These could be newer customers lured in by a promotion recently and have no reason to remain loyal, or on the other end, customers who are now takign advantage of their credit history to move to a different card that offers some incentives. Some effort should be invested to try and improve these customers loyalty. Perhaps by offering lower interest rates (or promotional rates), or some other incentives, these customers would remain.
Customers with no Balance or a Balance between 100-150K are also more likely to churn. For the higher Balance customers this suggests that they may be outgrowing the need for credit cards or finding alternate solutions. The Bank should look for ways to engage these customers with targeted marketing. Those with $0 Balance likely only have a credit card account (backed up by the majority of customers having a single NumOfProducts) and so are not particularly tied to the bank. The Bank should focus effort on getting these customers into additional product relationships or offer promotional incentives to place money into an account.
Finally, customers who are inactive are more likely to churn, possibly indicating dissatisfaction or indifference, leading to closing of the account. The Bank should encourage these customers to utilize their card or other bank services more often to keep customers like this from departing.
print('\n'*10)
Other models tried¶
Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, Dropout and LR (0.01)¶
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)
model_name = "Neural Network (128,64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.2, 0.1) + LR (0.01)"
#Initializing the model
adam_dropout_smote_model3 = Sequential()
adam_dropout_smote_model3.add(Dense(128,activation='relu',input_dim = X_train_smote.shape[1]))
adam_dropout_smote_model3.add(Dropout(0.8))
adam_dropout_smote_model3.add(Dense(64,activation='relu'))
adam_dropout_smote_model3.add(Dropout(0.4))
adam_dropout_smote_model3.add(Dense(32,activation='relu'))
adam_dropout_smote_model3.add(Dropout(0.2))
adam_dropout_smote_model3.add(Dense(16,activation='relu'))
adam_dropout_smote_model3.add(Dropout(0.2))
adam_dropout_smote_model3.add(Dense(8,activation='relu'))
adam_dropout_smote_model3.add(Dropout(0.1))
adam_dropout_smote_model3.add(Dense(1, activation = 'sigmoid'))
# use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)
# uncomment one of the following lines to define the metric to be used
metric = 'accuracy'
# metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
# compile the model with binary cross entropy as loss function and recall as the metric
adam_dropout_smote_model3.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
adam_dropout_smote_model3.summary()
Model: "sequential"
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāā ā Layer (type) ā Output Shape ā Param # ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā dense (Dense) ā (None, 128) ā 1,536 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout (Dropout) ā (None, 128) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_1 (Dense) ā (None, 64) ā 8,256 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_1 (Dropout) ā (None, 64) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_2 (Dense) ā (None, 32) ā 2,080 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_2 (Dropout) ā (None, 32) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_3 (Dense) ā (None, 16) ā 528 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_3 (Dropout) ā (None, 16) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_4 (Dense) ā (None, 8) ā 136 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_4 (Dropout) ā (None, 8) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_5 (Dense) ā (None, 1) ā 9 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāā
Total params: 12,545 (49.00 KB)
Trainable params: 12,545 (49.00 KB)
Non-trainable params: 0 (0.00 B)
adam_dropout_smote_history3 = adam_dropout_smote_model3.fit(
X_train_smote,y_train_smote,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data = (X_val,y_val))
Epoch 1/50 150/150 āāāāāāāāāāāāāāāāāāāā 2s 3ms/step - accuracy: 0.5878 - loss: 0.6655 - val_accuracy: 0.7315 - val_loss: 0.5162 Epoch 2/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7017 - loss: 0.5857 - val_accuracy: 0.7595 - val_loss: 0.5130 Epoch 3/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7089 - loss: 0.5835 - val_accuracy: 0.7405 - val_loss: 0.4941 Epoch 4/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7245 - loss: 0.5664 - val_accuracy: 0.7290 - val_loss: 0.4787 Epoch 5/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7235 - loss: 0.5569 - val_accuracy: 0.7200 - val_loss: 0.5031 Epoch 6/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7263 - loss: 0.5517 - val_accuracy: 0.7145 - val_loss: 0.4965 Epoch 7/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7404 - loss: 0.5383 - val_accuracy: 0.7285 - val_loss: 0.4749 Epoch 8/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7472 - loss: 0.5249 - val_accuracy: 0.7470 - val_loss: 0.4711 Epoch 9/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7461 - loss: 0.5336 - val_accuracy: 0.7255 - val_loss: 0.4975 Epoch 10/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7406 - loss: 0.5324 - val_accuracy: 0.7445 - val_loss: 0.4866 Epoch 11/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7417 - loss: 0.5329 - val_accuracy: 0.7530 - val_loss: 0.4776 Epoch 12/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7484 - loss: 0.5295 - val_accuracy: 0.7685 - val_loss: 0.4467 Epoch 13/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7463 - loss: 0.5338 - val_accuracy: 0.7310 - val_loss: 0.4677 Epoch 14/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7534 - loss: 0.5229 - val_accuracy: 0.7605 - val_loss: 0.4698 Epoch 15/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7474 - loss: 0.5373 - val_accuracy: 0.7390 - val_loss: 0.4886 Epoch 16/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7515 - loss: 0.5269 - val_accuracy: 0.7810 - val_loss: 0.4694 Epoch 17/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7524 - loss: 0.5244 - val_accuracy: 0.7050 - val_loss: 0.5172 Epoch 18/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7517 - loss: 0.5315 - val_accuracy: 0.7545 - val_loss: 0.4835 Epoch 19/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7517 - loss: 0.5261 - val_accuracy: 0.7585 - val_loss: 0.4726 Epoch 20/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7482 - loss: 0.5296 - val_accuracy: 0.7500 - val_loss: 0.5085 Epoch 21/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7527 - loss: 0.5224 - val_accuracy: 0.7480 - val_loss: 0.4813 Epoch 22/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7471 - loss: 0.5269 - val_accuracy: 0.7570 - val_loss: 0.4704 Epoch 23/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7495 - loss: 0.5265 - val_accuracy: 0.7920 - val_loss: 0.4641 Epoch 24/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7516 - loss: 0.5222 - val_accuracy: 0.7610 - val_loss: 0.4654 Epoch 25/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7484 - loss: 0.5225 - val_accuracy: 0.7715 - val_loss: 0.4747 Epoch 26/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7496 - loss: 0.5261 - val_accuracy: 0.7560 - val_loss: 0.4697 Epoch 27/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7462 - loss: 0.5256 - val_accuracy: 0.7890 - val_loss: 0.4576 Epoch 28/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7504 - loss: 0.5201 - val_accuracy: 0.7740 - val_loss: 0.4672 Epoch 29/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7580 - loss: 0.5120 - val_accuracy: 0.7460 - val_loss: 0.4816 Epoch 30/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7515 - loss: 0.5303 - val_accuracy: 0.7860 - val_loss: 0.4779 Epoch 31/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7500 - loss: 0.5186 - val_accuracy: 0.7605 - val_loss: 0.4611 Epoch 32/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7505 - loss: 0.5242 - val_accuracy: 0.7590 - val_loss: 0.4871 Epoch 33/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7528 - loss: 0.5291 - val_accuracy: 0.7725 - val_loss: 0.4795 Epoch 34/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7663 - loss: 0.5094 - val_accuracy: 0.7675 - val_loss: 0.4723 Epoch 35/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7540 - loss: 0.5148 - val_accuracy: 0.7840 - val_loss: 0.4772 Epoch 36/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7529 - loss: 0.5164 - val_accuracy: 0.7945 - val_loss: 0.4607 Epoch 37/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7552 - loss: 0.5207 - val_accuracy: 0.7840 - val_loss: 0.4550 Epoch 38/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7571 - loss: 0.5123 - val_accuracy: 0.7845 - val_loss: 0.4601 Epoch 39/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7600 - loss: 0.5150 - val_accuracy: 0.7585 - val_loss: 0.4599 Epoch 40/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7572 - loss: 0.5129 - val_accuracy: 0.7535 - val_loss: 0.5054 Epoch 41/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7488 - loss: 0.5352 - val_accuracy: 0.7655 - val_loss: 0.4940 Epoch 42/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7515 - loss: 0.5170 - val_accuracy: 0.7600 - val_loss: 0.4683 Epoch 43/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7510 - loss: 0.5168 - val_accuracy: 0.7870 - val_loss: 0.4544 Epoch 44/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7593 - loss: 0.5134 - val_accuracy: 0.7725 - val_loss: 0.4579 Epoch 45/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7530 - loss: 0.5202 - val_accuracy: 0.7835 - val_loss: 0.4668 Epoch 46/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7619 - loss: 0.5089 - val_accuracy: 0.7550 - val_loss: 0.4740 Epoch 47/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7507 - loss: 0.5162 - val_accuracy: 0.7545 - val_loss: 0.4833 Epoch 48/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7547 - loss: 0.5074 - val_accuracy: 0.7715 - val_loss: 0.4797 Epoch 49/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7523 - loss: 0.5227 - val_accuracy: 0.7995 - val_loss: 0.4472 Epoch 50/50 150/150 āāāāāāāāāāāāāāāāāāāā 0s 2ms/step - accuracy: 0.7517 - loss: 0.5260 - val_accuracy: 0.7820 - val_loss: 0.4573
Loss function
#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_dropout_smote_history3.history['loss'],adam_dropout_smote_history3.history['val_loss'])
y_train_pred = adam_dropout_smote_model3.predict(X_train_smote)
#Predicting the results using 0.5 as the threshold
y_train_pred = (y_train_pred > 0.5)
y_train_pred
299/299 āāāāāāāāāāāāāāāāāāāā 0s 744us/step
array([[False],
[ True],
[ True],
...,
[ True],
[False],
[ True]])
y_val_pred = adam_dropout_smote_model3.predict(X_val)
#Predicting the results using 0.5 as the threshold
y_val_pred = (y_val_pred > 0.5)
y_val_pred
63/63 āāāāāāāāāāāāāāāāāāāā 0s 596us/step
array([[False],
[False],
[ True],
...,
[False],
[False],
[ True]])
#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train_smote,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)
print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])
recall 0.779987 Name: Neural Network (128,64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.2, 0.1) + LR (0.01), dtype: float64 recall 0.737101 Name: Neural Network (128,64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.2, 0.1) + LR (0.01), dtype: float64
Classification report
print(classification_report(y_train_smote,y_train_pred))
precision recall f1-score support
0 0.78 0.80 0.79 4777
1 0.80 0.78 0.79 4777
accuracy 0.79 9554
macro avg 0.79 0.79 0.79 9554
weighted avg 0.79 0.79 0.79 9554
print(classification_report(y_val,y_val_pred))
precision recall f1-score support
0 0.92 0.79 0.85 1593
1 0.48 0.74 0.58 407
accuracy 0.78 2000
macro avg 0.70 0.77 0.72 2000
weighted avg 0.83 0.78 0.80 2000
Confusion matrix
#Calculating the confusion matrix
make_confusion_matrix(y_train_smote, y_train_pred)
#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)
Neural Network (64,32,16,8) with SMOTE, Adam & Dropout (0.8, 0.4, 0.2, 0.2, 0.2) + LR (0.003)¶
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)
batch_size_opt= 32
epochs_opt = 50
#Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.2)
print(X_train_smote.shape[1])
11
model_name = "Neural Network (64,32,16,8) with SMOTE, Adam & Dropout (0.8, 0.4, 0.2, 0.2, 0.2) + LR (0.003)"
#Initializing the model
adam_dropout_smote_model4 = Sequential()
adam_dropout_smote_model4.add(Dense(64,activation='relu',input_dim = X_train_smote.shape[1]))
adam_dropout_smote_model4.add(Dropout(0.8))
adam_dropout_smote_model4.add(Dense(32,activation='relu'))
adam_dropout_smote_model4.add(Dropout(0.4))
adam_dropout_smote_model4.add(Dense(16,activation='relu'))
adam_dropout_smote_model4.add(Dropout(0.2))
adam_dropout_smote_model4.add(Dense(8,activation='relu'))
adam_dropout_smote_model4.add(Dropout(0.2))
adam_dropout_smote_model4.add(Dense(1, activation = 'sigmoid'))
# use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam(learning_rate=0.003)
# uncomment one of the following lines to define the metric to be used
metric = 'accuracy'
# metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
# compile the model with binary cross entropy as loss function and recall as the metric
adam_dropout_smote_model4.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
adam_dropout_smote_model4.summary()
Model: "sequential"
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāā ā Layer (type) ā Output Shape ā Param # ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā dense (Dense) ā (None, 64) ā 768 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout (Dropout) ā (None, 64) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_1 (Dense) ā (None, 32) ā 2,080 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_1 (Dropout) ā (None, 32) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_2 (Dense) ā (None, 16) ā 528 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_2 (Dropout) ā (None, 16) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_3 (Dense) ā (None, 8) ā 136 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_3 (Dropout) ā (None, 8) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_4 (Dense) ā (None, 1) ā 9 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāā
Total params: 3,521 (13.75 KB)
Trainable params: 3,521 (13.75 KB)
Non-trainable params: 0 (0.00 B)
adam_dropout_smote_history4 = adam_dropout_smote_model4.fit(
X_train_smote,y_train_smote,
batch_size=batch_size_opt,
epochs=epochs,
verbose=1,
validation_data = (X_val,y_val))
Epoch 1/50 299/299 āāāāāāāāāāāāāāāāāāāā 2s 2ms/step - accuracy: 0.5472 - loss: 0.6852 - val_accuracy: 0.7180 - val_loss: 0.5740 Epoch 2/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.6804 - loss: 0.6245 - val_accuracy: 0.7025 - val_loss: 0.5768 Epoch 3/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.6957 - loss: 0.5968 - val_accuracy: 0.7140 - val_loss: 0.5534 Epoch 4/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7146 - loss: 0.5799 - val_accuracy: 0.7125 - val_loss: 0.5584 Epoch 5/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7206 - loss: 0.5689 - val_accuracy: 0.7170 - val_loss: 0.5222 Epoch 6/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7231 - loss: 0.5554 - val_accuracy: 0.7165 - val_loss: 0.5237 Epoch 7/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7407 - loss: 0.5478 - val_accuracy: 0.7310 - val_loss: 0.5103 Epoch 8/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7372 - loss: 0.5396 - val_accuracy: 0.7340 - val_loss: 0.5214 Epoch 9/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7473 - loss: 0.5305 - val_accuracy: 0.7300 - val_loss: 0.5289 Epoch 10/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7471 - loss: 0.5290 - val_accuracy: 0.7390 - val_loss: 0.5182 Epoch 11/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7486 - loss: 0.5269 - val_accuracy: 0.7435 - val_loss: 0.5032 Epoch 12/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7562 - loss: 0.5243 - val_accuracy: 0.7130 - val_loss: 0.5239 Epoch 13/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7541 - loss: 0.5268 - val_accuracy: 0.7285 - val_loss: 0.5274 Epoch 14/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7537 - loss: 0.5238 - val_accuracy: 0.7460 - val_loss: 0.5089 Epoch 15/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7493 - loss: 0.5198 - val_accuracy: 0.6990 - val_loss: 0.5281 Epoch 16/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7537 - loss: 0.5152 - val_accuracy: 0.7120 - val_loss: 0.5295 Epoch 17/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7523 - loss: 0.5145 - val_accuracy: 0.7400 - val_loss: 0.5180 Epoch 18/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7537 - loss: 0.5157 - val_accuracy: 0.7310 - val_loss: 0.5147 Epoch 19/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7604 - loss: 0.5087 - val_accuracy: 0.7370 - val_loss: 0.5086 Epoch 20/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7529 - loss: 0.5147 - val_accuracy: 0.7760 - val_loss: 0.5041 Epoch 21/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7536 - loss: 0.5139 - val_accuracy: 0.7100 - val_loss: 0.5160 Epoch 22/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7588 - loss: 0.5081 - val_accuracy: 0.7140 - val_loss: 0.4888 Epoch 23/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7519 - loss: 0.5091 - val_accuracy: 0.7090 - val_loss: 0.5244 Epoch 24/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7575 - loss: 0.5139 - val_accuracy: 0.7170 - val_loss: 0.5285 Epoch 25/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7538 - loss: 0.5118 - val_accuracy: 0.7100 - val_loss: 0.5079 Epoch 26/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7597 - loss: 0.5104 - val_accuracy: 0.7025 - val_loss: 0.5467 Epoch 27/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7614 - loss: 0.5065 - val_accuracy: 0.7055 - val_loss: 0.5337 Epoch 28/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7507 - loss: 0.5067 - val_accuracy: 0.7195 - val_loss: 0.5143 Epoch 29/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7521 - loss: 0.5099 - val_accuracy: 0.7830 - val_loss: 0.4965 Epoch 30/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7550 - loss: 0.5001 - val_accuracy: 0.7100 - val_loss: 0.5209 Epoch 31/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7551 - loss: 0.5022 - val_accuracy: 0.7330 - val_loss: 0.5158 Epoch 32/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7571 - loss: 0.5059 - val_accuracy: 0.7145 - val_loss: 0.4988 Epoch 33/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7588 - loss: 0.5047 - val_accuracy: 0.7275 - val_loss: 0.5053 Epoch 34/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7538 - loss: 0.5074 - val_accuracy: 0.7290 - val_loss: 0.4931 Epoch 35/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7620 - loss: 0.5045 - val_accuracy: 0.7780 - val_loss: 0.4965 Epoch 36/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7606 - loss: 0.4981 - val_accuracy: 0.7785 - val_loss: 0.4969 Epoch 37/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7549 - loss: 0.5117 - val_accuracy: 0.7845 - val_loss: 0.5040 Epoch 38/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7620 - loss: 0.4974 - val_accuracy: 0.7145 - val_loss: 0.5134 Epoch 39/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7617 - loss: 0.5004 - val_accuracy: 0.7360 - val_loss: 0.5017 Epoch 40/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7617 - loss: 0.4968 - val_accuracy: 0.7230 - val_loss: 0.4860 Epoch 41/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7561 - loss: 0.5061 - val_accuracy: 0.7910 - val_loss: 0.4901 Epoch 42/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7531 - loss: 0.5032 - val_accuracy: 0.7465 - val_loss: 0.4763 Epoch 43/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7556 - loss: 0.5039 - val_accuracy: 0.7395 - val_loss: 0.5131 Epoch 44/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7570 - loss: 0.5017 - val_accuracy: 0.7195 - val_loss: 0.5359 Epoch 45/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7629 - loss: 0.4977 - val_accuracy: 0.7000 - val_loss: 0.5080 Epoch 46/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7555 - loss: 0.5025 - val_accuracy: 0.7395 - val_loss: 0.4976 Epoch 47/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7567 - loss: 0.4999 - val_accuracy: 0.7755 - val_loss: 0.5085 Epoch 48/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7583 - loss: 0.5013 - val_accuracy: 0.7855 - val_loss: 0.4902 Epoch 49/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7553 - loss: 0.5047 - val_accuracy: 0.7845 - val_loss: 0.4891 Epoch 50/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - accuracy: 0.7629 - loss: 0.5028 - val_accuracy: 0.8020 - val_loss: 0.4726
Loss function
#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_dropout_smote_history4.history['loss'],adam_dropout_smote_history4.history['val_loss'])
y_train_pred = adam_dropout_smote_model4.predict(X_train_smote)
#Predicting the results using 0.5 as the threshold
y_train_pred = (y_train_pred > 0.5)
y_train_pred
299/299 āāāāāāāāāāāāāāāāāāāā 0s 704us/step
array([[False],
[ True],
[ True],
...,
[ True],
[False],
[ True]])
y_val_pred = adam_dropout_smote_model4.predict(X_val)
#Predicting the results using 0.5 as the threshold
y_val_pred = (y_val_pred > 0.5)
y_val_pred
63/63 āāāāāāāāāāāāāāāāāāāā 0s 544us/step
array([[False],
[False],
[ True],
...,
[False],
[False],
[ True]])
#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train_smote,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)
print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])
recall 0.753192 Name: Neural Network (64,32,16,8) with SMOTE, Adam & Dropout (0.8, 0.4, 0.2, 0.2, 0.2) + LR (0.003), dtype: float64 recall 0.692875 Name: Neural Network (64,32,16,8) with SMOTE, Adam & Dropout (0.8, 0.4, 0.2, 0.2, 0.2) + LR (0.003), dtype: float64
Classification report
print(classification_report(y_train_smote,y_train_pred))
precision recall f1-score support
0 0.77 0.84 0.80 4777
1 0.82 0.75 0.79 4777
accuracy 0.80 9554
macro avg 0.80 0.80 0.80 9554
weighted avg 0.80 0.80 0.80 9554
print(classification_report(y_val,y_val_pred))
precision recall f1-score support
0 0.91 0.83 0.87 1593
1 0.51 0.69 0.59 407
accuracy 0.80 2000
macro avg 0.71 0.76 0.73 2000
weighted avg 0.83 0.80 0.81 2000
Confusion matrix
#Calculating the confusion matrix
make_confusion_matrix(y_train_smote, y_train_pred)
#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)
Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, Dropout, Adjusting LR and Batch size¶
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)
model_name = "Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.2) + LR (0.01) + Batch (32)"
#Initializing the model
adam_dropout_smote_model2b = Sequential()
adam_dropout_smote_model2b.add(Dense(64,activation='relu',input_dim = X_train_smote.shape[1]))
adam_dropout_smote_model2b.add(Dropout(0.8))
adam_dropout_smote_model2b.add(Dense(32,activation='relu'))
adam_dropout_smote_model2b.add(Dropout(0.4))
adam_dropout_smote_model2b.add(Dense(16,activation='relu'))
adam_dropout_smote_model2b.add(Dropout(0.2))
adam_dropout_smote_model2b.add(Dense(8,activation='relu'))
adam_dropout_smote_model2b.add(Dropout(0.2))
adam_dropout_smote_model2b.add(Dense(1, activation = 'sigmoid'))
# use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)
# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
# compile the model with binary cross entropy as loss function and recall as the metric
adam_dropout_smote_model2b.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
adam_dropout_smote_model2b.summary()
Model: "sequential"
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāā ā Layer (type) ā Output Shape ā Param # ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā dense (Dense) ā (None, 64) ā 768 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout (Dropout) ā (None, 64) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_1 (Dense) ā (None, 32) ā 2,080 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_1 (Dropout) ā (None, 32) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_2 (Dense) ā (None, 16) ā 528 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_2 (Dropout) ā (None, 16) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_3 (Dense) ā (None, 8) ā 136 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dropout_3 (Dropout) ā (None, 8) ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā⤠ā dense_4 (Dense) ā (None, 1) ā 9 ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāā
Total params: 3,521 (13.75 KB)
Trainable params: 3,521 (13.75 KB)
Non-trainable params: 0 (0.00 B)
adam_dropout_smote_history2b = adam_dropout_smote_model2b.fit(
X_train_smote,y_train_smote,
batch_size=32,
epochs=epochs,
verbose=1,
validation_data = (X_val,y_val))
Epoch 1/50 299/299 āāāāāāāāāāāāāāāāāāāā 2s 2ms/step - loss: 0.6699 - recall: 0.5114 - val_loss: 0.5946 - val_recall: 0.7666 Epoch 2/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.6169 - recall: 0.7236 - val_loss: 0.5770 - val_recall: 0.7764 Epoch 3/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5914 - recall: 0.7060 - val_loss: 0.5366 - val_recall: 0.7912 Epoch 4/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5888 - recall: 0.7379 - val_loss: 0.5668 - val_recall: 0.8452 Epoch 5/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5746 - recall: 0.7198 - val_loss: 0.5906 - val_recall: 0.8157 Epoch 6/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5612 - recall: 0.7184 - val_loss: 0.5411 - val_recall: 0.8354 Epoch 7/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5615 - recall: 0.7130 - val_loss: 0.5524 - val_recall: 0.8575 Epoch 8/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5483 - recall: 0.7159 - val_loss: 0.5149 - val_recall: 0.8231 Epoch 9/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5520 - recall: 0.7134 - val_loss: 0.5206 - val_recall: 0.8354 Epoch 10/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5465 - recall: 0.7361 - val_loss: 0.4860 - val_recall: 0.7322 Epoch 11/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5398 - recall: 0.7232 - val_loss: 0.5323 - val_recall: 0.7985 Epoch 12/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5501 - recall: 0.7328 - val_loss: 0.5591 - val_recall: 0.8157 Epoch 13/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5431 - recall: 0.7303 - val_loss: 0.5660 - val_recall: 0.8305 Epoch 14/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5482 - recall: 0.7276 - val_loss: 0.5355 - val_recall: 0.8673 Epoch 15/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5363 - recall: 0.7521 - val_loss: 0.5096 - val_recall: 0.8477 Epoch 16/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5393 - recall: 0.7474 - val_loss: 0.5106 - val_recall: 0.8059 Epoch 17/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5374 - recall: 0.7400 - val_loss: 0.5304 - val_recall: 0.8231 Epoch 18/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5407 - recall: 0.7383 - val_loss: 0.5167 - val_recall: 0.8501 Epoch 19/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5299 - recall: 0.7545 - val_loss: 0.5078 - val_recall: 0.8182 Epoch 20/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5370 - recall: 0.7340 - val_loss: 0.4968 - val_recall: 0.7224 Epoch 21/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5402 - recall: 0.7292 - val_loss: 0.5176 - val_recall: 0.8329 Epoch 22/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5257 - recall: 0.7416 - val_loss: 0.4756 - val_recall: 0.7543 Epoch 23/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5391 - recall: 0.7345 - val_loss: 0.5333 - val_recall: 0.8624 Epoch 24/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5376 - recall: 0.7449 - val_loss: 0.5289 - val_recall: 0.8722 Epoch 25/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5484 - recall: 0.7580 - val_loss: 0.4934 - val_recall: 0.7887 Epoch 26/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5411 - recall: 0.7430 - val_loss: 0.5221 - val_recall: 0.8280 Epoch 27/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5449 - recall: 0.7298 - val_loss: 0.4899 - val_recall: 0.7518 Epoch 28/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5452 - recall: 0.7362 - val_loss: 0.5094 - val_recall: 0.8182 Epoch 29/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5320 - recall: 0.7208 - val_loss: 0.5611 - val_recall: 0.8378 Epoch 30/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5331 - recall: 0.7290 - val_loss: 0.4912 - val_recall: 0.7027 Epoch 31/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5316 - recall: 0.7329 - val_loss: 0.4894 - val_recall: 0.8206 Epoch 32/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5408 - recall: 0.7319 - val_loss: 0.5021 - val_recall: 0.8108 Epoch 33/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5383 - recall: 0.7396 - val_loss: 0.4913 - val_recall: 0.7396 Epoch 34/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5340 - recall: 0.7200 - val_loss: 0.5059 - val_recall: 0.7420 Epoch 35/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5355 - recall: 0.7367 - val_loss: 0.5086 - val_recall: 0.7445 Epoch 36/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5324 - recall: 0.7429 - val_loss: 0.5275 - val_recall: 0.8231 Epoch 37/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5291 - recall: 0.7265 - val_loss: 0.5237 - val_recall: 0.7838 Epoch 38/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5234 - recall: 0.7638 - val_loss: 0.5275 - val_recall: 0.8526 Epoch 39/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5306 - recall: 0.7611 - val_loss: 0.4727 - val_recall: 0.8010 Epoch 40/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5344 - recall: 0.7553 - val_loss: 0.4775 - val_recall: 0.7273 Epoch 41/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5410 - recall: 0.6939 - val_loss: 0.4986 - val_recall: 0.8059 Epoch 42/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5335 - recall: 0.7168 - val_loss: 0.4970 - val_recall: 0.8231 Epoch 43/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5330 - recall: 0.7431 - val_loss: 0.4850 - val_recall: 0.7961 Epoch 44/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5290 - recall: 0.7705 - val_loss: 0.5077 - val_recall: 0.8182 Epoch 45/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5226 - recall: 0.7556 - val_loss: 0.4974 - val_recall: 0.8206 Epoch 46/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5263 - recall: 0.7578 - val_loss: 0.5254 - val_recall: 0.8157 Epoch 47/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5274 - recall: 0.7476 - val_loss: 0.5203 - val_recall: 0.8428 Epoch 48/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5366 - recall: 0.7580 - val_loss: 0.4795 - val_recall: 0.7887 Epoch 49/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5366 - recall: 0.7382 - val_loss: 0.4790 - val_recall: 0.7543 Epoch 50/50 299/299 āāāāāāāāāāāāāāāāāāāā 0s 1ms/step - loss: 0.5422 - recall: 0.7380 - val_loss: 0.4900 - val_recall: 0.7936
Loss function
#Plotting Train Loss vs Validation Loss
plot_train_vs_val_loss(adam_dropout_smote_history2b.history['loss'],adam_dropout_smote_history2b.history['val_loss'])
y_train_pred = adam_dropout_smote_model2b.predict(X_train_smote)
#Predicting the results using 0.5 as the threshold
y_train_pred = (y_train_pred > 0.5)
y_train_pred
299/299 āāāāāāāāāāāāāāāāāāāā 0s 670us/step
array([[False],
[ True],
[ True],
...,
[ True],
[False],
[ True]])
y_val_pred = adam_dropout_smote_model2b.predict(X_val)
#Predicting the results using 0.5 as the threshold
y_val_pred = (y_val_pred > 0.5)
y_val_pred
63/63 āāāāāāāāāāāāāāāāāāāā 0s 561us/step
array([[False],
[False],
[ True],
...,
[False],
[False],
[ True]])
#Save metrics for later comparison
training_metrics.loc[model_name] = recall_score(y_train_smote,y_train_pred)
validation_metrics.loc[model_name] = recall_score(y_val,y_val_pred)
print(training_metrics.loc[model_name])
print(validation_metrics.loc[model_name])
recall 0.840695 Name: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.2) + LR (0.01) + Batch (32), dtype: float64 recall 0.793612 Name: Neural Network (64,32,16,8) with SMOTE,Adam & Dropout (0.8, 0.4, 0.2, 0.2) + LR (0.01) + Batch (32), dtype: float64
Classification report
print(classification_report(y_train_smote,y_train_pred))
precision recall f1-score support
0 0.82 0.74 0.78 4777
1 0.76 0.84 0.80 4777
accuracy 0.79 9554
macro avg 0.79 0.79 0.79 9554
weighted avg 0.79 0.79 0.79 9554
print(classification_report(y_val,y_val_pred))
precision recall f1-score support
0 0.93 0.71 0.81 1593
1 0.41 0.79 0.54 407
accuracy 0.73 2000
macro avg 0.67 0.75 0.68 2000
weighted avg 0.83 0.73 0.75 2000
Confusion matrix
#Calculating the confusion matrix
make_confusion_matrix(y_train_smote, y_train_pred)
#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)